Part 1 of this series discussed endotoxin contamination in a WFI system. During the investigation, we noticed something strange in our historical data review. A couple city water samples didn’t have their expected endotoxin. Part 2 of this series looked into raw data for those tests. For one of those two tests, endotoxin levels were just below the limit of detection threshold. That made sense. I ended part 2 saying “I saw _______ ”. I wasn’t leaving you with a blank to fill in. I literally saw a flat line. What’s so interesting about a flat line? Quick recap- The lysate we used for testing (i.e. purified horseshoe crab blood), turns yellow over time. It turns yellow faster when combined with endotoxin. The test measures how fast samples turn yellow by collecting yellow intensity readings every couple minutes. After most tests, the raw data graphs look like this: Part 2 really goes into detail on this, but the squares are graphs representing each well of a 96 well plate. The right half of the plate didn’t have anything in the wells, so the raw data graphs look like empty boxes. They're actually flat lines running along the x-axis. The raw data for the 2nd atypical city water day looked like this: Tests wells with lysate, but no endotoxin, should have graphs like this: For the test in question, the graphs looked like lysate was never added to them, like this: Part 3 of this endotoxin series starts here. We found a single assay that looked like lysate wasn’t added to test and negative control wells. If that was truly the case, the samples were never tested, but the assay was guaranteed to have all results meet specification. The official test report can't detect this error.
So what do we do next? A bunch of things simultaneously. Identify our scope. It would’ve been easy to write this off completely. The raw data wasn’t used for any official purpose. We even reached out to the equipment supplier and they couldn’t support any conclusions from the raw data. According to them, if all parameters in the final report met requirements, the assay was acceptable. It also would’ve been easy to call this a one-off assay with (potentially) invalid results. We could have written up a low-impact non-conformance report and called it a day. But I’m proud my company supported the search for, and reporting of, everything impacted by this issue. Further review determined a single analyst was responsible for 3 months of assays with this pattern. We immediately considered them all invalid. Lysate was not used in the test wells of:
Damage Control The clock was ticking. Per 21 CFR 314.81(b)(1), the site was required to submit a Field Alert Report (FAR) to the FDA within 3 days. We had information concerning significant quality problems with distributed drug product. We had dozens of lots on the market without a valid, required relase test. To make matters worse, this site received a warning letter for endotoxin issues less than a decade prior to this event. It was all hands on deck in the endotoxin lab. We had 3 days to generate valid tests for all impacted lots. We tested retains for all of them. We barely had enough test equipment to complete all samples that quickly, but we somehow got it done. A non-conformance event was opened to address the impact of the water sampling that couldn’t be replicated. We also had to re-do equipment validations. We got through it, and I feel as confident as I can be in the quality decisions that were made during this time. Handle a Human Resources Nightmare. Officially, the analyst consistently repeated an error in their technique. This error somehow developed after months of testing correctly. The analyst added lysate to standard and positive controls wells first. These wells are supposed to have endotoxin - they require a lysate reaction for the final report. The analyst claimed adding lysate to these wells first helped prevent contamination (that in itself is faulty logic). Then, the analyst “forgot” to add lysate to the test wells. There were other raw data inconsistencies that contradict the official story. A variety of different colored products were tested over these three months. These products all have different “baselines” - the first reading when the equipment starts measuring yellow intensity of the sample. However, in assays where the analyst claimed to test these products, the baseline readings for the sample positive controls (where we know lysate was added) were the same as the baselines for water positive controls. The baseline should have been much higher for a colored product. To me- that meant the analyst never even attempted to test the product. I dug into all assays this analyst performed. I also checked on multiple assays performed by other testers to confirm this wasn’t a department wide issue. I came to the following conclusion: The analyst knowingly tried to cheat the test to prevent “complications”
I wasn’t involved in the HR and management discussions. I believe they agreed with my conclusions, but they officially aligned with the analysts’ explanation of events. I think they were stuck:
Prevent this from ever happening again. We made raw data review an official process moving forward. Every analyst was trained on what to look for when performing and reviewing assays. There were strange nuances though. For example, if a bubble formed during pipetting, it could impact the baseline reading and cause a flat line graph. There were ways to tell if that was the case, but impossible to define in an SOP. Clearly the analyst never should have made this error. I think the stress of the job contributed to it. The job occurred on 2nd shift. This shift was understaffed and often dealt with time sensitive samples. The samples could come at any time during the shift. A minor issue with an assay at the end of the shift could require the analyst to work unpaid (salaried) overtime through the night. There was no 3rd shift in the lab to help on those late nights. The temptation to cut corners proved too much. Soon after this event, a full-time 3rd shift position was created. It was extremely comforting knowing critical samples could be handled as a team. It removed a lot of the corner-cutting temptation and significantly reduced the stress on the job in general. This was such a strange event it never would've repeated regardless of the actions we took. But the Lab became a better place because of those actions.
0 Comments
I apologize in advance. This post is BORING (Although I’m proud of the cliff hanger ending). We get into the weeds of kinetic-chromagenic test equipment and how it determines endotoxin levels in a sample. This background is needed to understand the craziness of what happened. This is the 2nd part of a 3 part series. Part 1 discussed WFI system endotoxin contamination. The endotoxin came from pre-treatment water. We always had some level of endotoxin in the city water, but we wanted to know why we didn’t see it on two particular days. I believed I could answer that question by looking at the “unofficial” raw data from the test. The test measures endotoxin in a cool way. When the purified horseshoe crab blood (lysate) mixed with endotoxin, it turned the solution yellow. The equipment measured the yellow intensity throughout the test. The faster the samples turned yellow, the more endotoxin was there. When a sample reached a certain threshold, we said it “reacted” We tested with a 96 well plate like this: We added samples + lysate to each of those wells. When the test was done, the plate had a bunch of different shades of yellow. I made a mock-up in excel of what it kinda looked like. This shows what a plate with 9 samples looked like when it was done. (A plate could fit 20 samples if needed): And this is what those shades represented: An official report was generated after each test. Results from the report were really the only thing management cared about. Those results allowed us to release a lot or confirm WFI was acceptable for use. But what else was on that report?
Some measurements had to line up for the entire test plate to be valid. A test plate was only valid if all the following occurred:
If we had a problem with any of those, the entire assay had to be repeated. Other weird stuff could happen too. Things like hot wells (when a single well reacted when it wasn’t expected to) required us to repeat individual tests. It was miserable when I got a hot well at the end of my shift. I’d be stuck at work for another hour. All information discussed above was on each final test report. What else could possibly be gathered from the raw data? During the test, every 2 minutes, the machine measured the intensity of yellow in each test well. The raw data contained all those measurements, plus a graph for each of those measurements over time. When I tested, I watched these graphs in real time hoping nothing looked out of the ordinary. The examples below show what those graphs looked like over the course of an assay. Remember, each little square is a graph for a corresponding well on the plate. 6 minutes in: The most concentrated standard started to turn yellow. Those tiny marks on the fifth row down represent a slight elevation in yellowness compared to the first measurement taken. No changes are detected for the other wells. At this time, the plotted lines are indistinguishable from the X axis for those wells. 16 minutes in: The most concentrated standard has reached the reaction threshold. It will continue to increase in yellow intensity (which explains why it looks darker when the assay is complete), but any further change is irrelevant to the assay. The 2nd most concentrated standard is almost at the threshold. The color change for the 3rd standard and positive controls are just starting to be detected. 36 minutes in: Every standard and control has reached the threshold except for the most dilute standard. What’s interesting here- we see all the tests and negative controls start to change color. That’s right! Lysate will turn yellow on its own over time, even if no endotoxin is there! 46 minutes in: The assay is complete. The final graph doesn’t look much different than the 36 minute graph: Phew! The background is all laid out. I can finally explain what I expected to see for those 2 city water samples with no endotoxin. I thought the graphs for the duplicate test wells would look like the circled wells here: To me, those graphs say some endotoxin was there, it just wasn’t enough to meet the threshold. The line rose faster than the negative control samples, but the endotoxin level was just below the most diluted standard we tested.
For one of the 2 days in question, that was exactly what I saw. It completely made sense. Endotoxin levels were just a bit lower than the low end of our normal range. If the lowest standard had 0.05 EU/mL, this city water sample probably had 0.03 EU/mL. No big deal. On the other day in question, I saw . <-- I fill in that blank in Part 3. (If you really paid attention, you’ll know how this isn’t a cliff hanger). |