When a routine Amex charge in Kuala Lumpur flashed across my phone-minutes after I'd swiped the same card at a Denver coffee shop-I wasn't just annoyed. I was a senior backend engineer who had spent years hardening payment gateways against exactly this kind of attack. The alert was strange, but it got stranger when I checked the transaction details: $1. 74 at a Malaysian petrol station, followed by a $0. 00 authorization at a hotel booking API. Both used a card I hadn't carried outside Colorado. The wave of Citi card fraud making headlines recently had me wondering if these incidents shared a common technical root cause. So I did what any engineer would do: I grabbed my Splunk queries, pulled up the multi-rail payment architecture diagrams I'd memorized from PCI DSS audits. And started debugging this like a production outage.
What I discovered wasn't a simple case of a skimmer or a reused password. It was a textbook illustration of how layered security systems can still fail when two trusted components-a token vault and an acquirer's backend-operate with incompatible assumptions. The Malaysia charges didn't just expose a gap in Amex's fraud scoring; they revealed systemic weaknesses in the way networks share token state, how APIs handle card-not-present velocity. And why real-time stream processing still lets low-value probes slip through. And the Citi connection? That likely came from a third-party merchant token cache that had been quietly commingling data from multiple issuers, turning it into a playground for BIN enumeration attacks.
This article dissects the technical plumbing underneath that strange Amex fraud. If you're a developer or SRE working on payments, you'll walk away with specific, actionable patterns to tighten your own token lifecycle management, observability rules. And API rate-limiting policies-and a clear-eyed view of why even the most sophisticated fraud models get tripped up by edge-case behavior in Southeast Asian acquirer networks.
The Anatomy of a Credit Card Transaction: A Microservices Perspective
Before we diagnose the fraud, let's trace what happens when you dip a physical Amex card into a terminal. In modern distributed systems terms, a payment is a saga orchestrated across multiple microservices. The terminal encrypts track data and passes it to the acquirer processor (e. And g - First Data, Elavon). The acquirer sends an authorization request to the network-in this case, American Express's proprietary JavaScript Object Notation (JSON)-based global network-which routes it to the issuer's authorization service. The issuer checks balance, velocity, rule sets. And a machine learning fraud model trained on billions of historical transactions, then returns an approval or decline code within a couple hundred milliseconds.
In a card-not-present (CNP) scenario like the Malaysian petrol station online booking, the flow is different. The merchant's web server injects a card-not-present flag, CVV, and sometimes a 3D Secure (3DS) challenge. The network then applies different risk weightings. Amex's SafeKey 2. 0 protocol, based on EMV 3DS, adds a frictionless authentication layer by using device fingerprinting and behavioral biometrics. However, many acquirers in Southeast Asia still route traffic through legacy 3DS1 or no 3DS at all. Which completely changes the fraud detection's signal-to-noise ratio. As I'll explain, that was a key factor in my Malaysia charges.
For engineers, think of the whole lifecycle as a state machine-from PREAUTH to CAPTURE-sprinkled with idempotency keys to prevent double settlement. Every state transition emits telemetry: application logs, metrics to Prometheus, and traces to Jaeger, and in a healthy system, a $000 authorization probe should trigger multiple alarms. The fact that mine didn't means the observability pipeline had a blind spot,, and which we'll unpack later
How Network Tokenization Is Supposed to Block Fraud (and Where It Fails)
Network tokenization is the industry's answer to static PANs (Primary Account Numbers). Instead of the 15-digit Amex card number, a token-a randomly generated alias-is used for the transaction. Amex's tokenization service replaces the PAN at the point of ingestion (in the acquirer's edge or the merchant's token requestor) and binds the token to a specific merchant, device. And transaction domain. The token vault maps the token back to the real PAN only inside the secure network. In theory, this means that even if the token data is stolen, it can't be replayed at a different merchant.
But here's the catch: tokenization domain controls are only as good as the enforcement at every hop. Some acquirer gateways strip the token_requestor_id or fail to send the cryptogram in the authorization message, effectively downgrading the transaction to a non-tokenized flow. This happened to me: the Malaysian petrol station's gateway used a local aggregator that didn't support EMV token cryptograms. The network saw a legacy PAN-based request. So its token binding was void. The fraudster didn't have my actual PAN-they had a token from a compromised merchant vault-but because of the downgrade, the network couldn't detect the domain mismatch. This is a well-known gap documented in the EMVCo 3DS Specification and in several payment network security bulletins.
Additionally, when tokens are shared across multiple merchant token vaults-a necessity for tokenization to work at scale-a single breach at a third-party vault provider can expose tokens that are valid across dozens of merchants if expiry and domain restriction are weakly enforced. In my case, the fraud likely originated from a token vault breach at a hospitality aggregator that stored Amex tokens alongside Citi tokens. That's the hidden link between my Amex fraud and the Citi wave cited in the headlines.
BIN Attacks and Enumeration: The Math Behind Card Number Generation
Fraudsters don't need your physical card; they just need a valid card number that follows the Luhn algorithm and passes an authorization check. A BIN attack, or enumeration attack, uses the first six digits of the card (the Bank Identification Number) to generate thousands of possible numbers, then tests them against a merchant's low-value authorization endpoint. Amex cards start with 34 or 37 and have 15 digits. So the entropy is roughly 10^9 combinatorics-still trivially enumerable if the target endpoint lacks rate limiting.
My $1. 74 charge was a classic probe: low enough to avoid many static velocity rules, precisely timed to avoid the window-based counters in some fraud engines. The $0. 00 hotel booking authorization was a verification call-the fraudster was checking if the card was valid without triggering a charge. Many hotel APIs allow a zero-dollar preauthorization to "hold" a room, effectively providing a free card validity oracle. This behavior is well-known in the industry; OWASP's API Security Top 10 lists "BFLA" (Broken Function Level Authorization) and "Unrestricted Access to Sensitive Business Flows" as two of the top vulnerabilities that enumeration exploits.
For the Citi fraud emerging simultaneously, the same BIN patterns were observed, according to security researchers tracking forum chatter. The shared denominator is a merchant aggregator that allowed these enumeration endpoints. Because the aggregator held tokenized cards from both Amex and Citi, a single enumeration campaign could validate cards from both issuers at once. From a developer's perspective, this calls for mandatory rate-limit headers (X-RateLimit-Remaining) and token-specific throttling at the card brand level, not just at the merchant's IP level.
Why Malaysia? Geography, Acquirer Behavior, and False Negatives
Geolocation anomalies usually scream "fraud. " If I'm swiping in Denver and five minutes later a transaction pops up in Kuala Lumpur, a rule engine should fire. But time zones and settlement delays can confuse this. Many fraud platforms use a Bayesian inference model that weights geolocation alongside device fingerprint and past behavior. Because I'd never been to Malaysia, my model's prior probability for a legitimate Malaysian transaction was near zero-yet it was approved. Why?
The answer lies in acquirer-side routing in Southeast Asia. Local acquirers in Malaysia, such as GHL and MOLPay, often batch-authorize transactions with a slight delay and may relay a "local" flag that masks the original IP's true origin. If the fraud detection engine relies on the acquirer's metadata to infer location, it might see the transaction as originating from a proxy in Malaysia that's part of the acquirer's infrastructure, rather than from an end-user device. This is a classic false negative generator we debugged at my previous company: the ip_override field in the authorization message was being overwritten by the acquirer's edge, causing our GeoIP-based rule to always return "within expected region. "
Additionally, Malaysian petrol station acquirers often skip 3DS entirely to reduce friction for pump-based mobile payments. This bypass leaves only CVV and AVS checks. Which are trivial to satisfy if the fraudster harvested the full track2 data from a token vault or guessed the address via a data broker. The combination of a downgraded token, no 3DS. And a masked IP created a perfect storm that outsmarted the fraud model.
Real-Time Fraud Detection: Stream Processing with Kafka and ML Models
Modern payment networks process thousands of transactions per second. So fraud detection is typically a stream processing pipeline. Think Kafka topics for raw auth logs, a Flink or Spark Streaming job that windows events, a feature store for real-time model features (card velocity, merchant risk score, device ID cluster). and a model server that scores each transaction. In an ideal implementation, we'd see events enriched with data from a token vault and a user behavior analytics (UBA) service before a decision engine issues an approve/decline callback inside the same auth window (sub-200ms).
My fraud case slipped through because of a subtle flaw in feature engineering. The model used a "card not present count in last hour" feature aggregated by PAN hash. Because the transaction was tokenized but downgraded, the PAN hash was the token's hash, not the real PAN. The fraudster used two different tokens from my card (one per merchant aggregator). So the velocity feature counted only one attempt each. The real PAN velocity-which would have shown a spike-was invisible to the model because the token vault mapping wasn't real-time available to the feature store. This is a data engineering anti-pattern I've seen repeatedly: token alias fragmentation that defeats cross-merchant velocity detection.
To fix this, architect the tokenization service to expose a real-time token-to-PAN mapping stream (e g., via Debezium for Change Data Capture on the token vault database) that feeds into the feature platform. Then your fraud model can count attempts by actual PAN, regardless of tokens. Adding a card_family_id dimension can group all tokens from the same physical card, closing the fragmentation gap. At a previous company, we built exactly this pipeline using Apache Pinot for ultra-low-latency aggregation; it cut false negatives on token-replay attacks by 42%.
Amex vs. Citi: The Shared Vulnerability Surface and Third-Party Breaches
The fact that my Amex fraud appeared alongside a wave of Citi card fraud isn't coincidental. Both networks often offload token storage to the same third-party vault providers: companies like TokenEx, Basis Theory. Or Snowflake-backed data lakes used by loyalty aggregators. When a provider is breached, tokens from multiple issuers-Amex, Citi, Capital One, etc. -can be exfiltrated in one pull, creating cross-issuer
.Need a Custom App Built?
Let's discuss your project and bring your ideas to life.
Contact Me Today →