Situation
Sensibill, a fintech, put receipt and spending features inside the mobile apps of banks. It had about thirty clients, and each contract charged a penalty for any started hour of outage, counted from the first minute.
I joined in May 2017 as a junior developer, to automate the testing of the iOS, Android and web
apps and to keep the machines those tests ran on. Within months I also joined the platform team.
There were three of us, running about 180 Linux virtual machines across a few regions on IBM Cloud
and AWS.
Approach
Two test frameworks. I wrote them in Java with TestNG. One covered the native apps, Objective-C
and Swift on iOS, Java and Kotlin on Android, on Appium. The other covered the two web apps, in
Chrome, Firefox and Edge on macOS and Linux, on Selenium.
The device lab. I kept the machines the tests ran on. Mac minis ran the
native macOS builds, Chrome on Mac and the iOS simulators. Bitrise agents took a mix of jobs. About
thirty physical phones, mostly Android, ran on Smartphone Test Farm, and Mac minis and Linux
machines ran the Android emulators. Jenkins and Bitrise could then provision a matrix of devices on
the fly, across about five years of Android versions, back to 4.4 (KitKat) as the contracts
required. The team could also borrow the phones on demand.
One environment for everyone. I put the whole back end in Docker. New developers and the CI
started from the same environment.
Load tests on beta. I organised load tests with JMeter on the beta environments, up to 300,000
concurrent connections. They found a bottleneck in the load balancer before production traffic
reached it.
The move from IBM Cloud to AWS. With the platform team, I helped move the product’s back end
while both clouds stayed live: receipt processing, storage kept compliant with GDPR, transcripts,
and the integrations with banks. The move included terabytes of customer data, from object storage
to S3. I took part in the discussions that set how the migration would run.
Decision
We ran both clouds live during the move.
Turned down: a cut-over window, which under these contracts would start the penalty clock
Night incidents. On call, I caught a product issue in production at night more than once. Each time I kept the feature serving with fallbacks on the stored data. Pages rendered correctly, though images were not up to the minute. Every client stayed within its SLA until the fix in the morning.
Decision
Keep serving correct but slightly stale results overnight, and fix the issue in daylight.
Turned down: taking the feature offline until a fix, which would have started the penalties
Result
The mobile apps and the web apps each had one test framework, and the CI could run them on a device matrix it provisioned on the fly. The product moved to AWS and kept serving. The overnight fallbacks avoided six-figure SLA penalties. I left in June 2019 as a DevOps engineer, with Linux, cloud and on-call as daily work.
Next time
I would sanitize the data before any load test, and I would make the separation of environments that hold hot or real data a rule that comes before everything else. I would keep the CI services, but move the weight of testing closer to the code, and keep end-to-end tests for the global experiences worth verifying as a whole.