IM Doc pt5

Aug. 11th, 2021 09:35 pm
temporaryreality: (Default)
[personal profile] temporaryreality
 Please see the updated link in the topmost IM-Doc related post on this blog to find the current discussion location.

A Document Maven Looks at the Pfizer Vaccine Paper in the New England Journal of Medicine

By Lambert Strether of Corrente

As it turns out, I have narrow but deep expertise relevant to the discussion of the New England Journal of Medicine (NEJM) paper on the Pfizer vaccine study, “Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine” (“Safety and Efficacy”), so I thought I’d weigh in, albeit a little late, on the protocols and “exclusion criteria” discussion.

In a past life, several careers ago, I consulted to various large firms, among them medical publishers, on complex document structures; my deliverable would be a formal definition of such structures, plus documentation. For example, and without getting into the weeds on syntax, the structure of all posts at Naked Capitalism includes editorial elements like: a headline (one, required), author (at least one, with an optional bio), and then a long string of running text elements (paragraphs, images, tables, lists, and so on) in any order and any amount greater than one. There are also elements that occur within running text, like emphasis (italic), or links (with their URLs). Run your eye up and down this page, and you will see what I’m talking about. (No, this is not an exhaustive list, and there are plenty of decisions to me made about representation; but let’s not go into those weeds.) I loved that career, because it allowed me to combine my skills as a humanities major with exacting technical work.

Publishers liked having formal definitions of their document structures because (again leaving out syntax) it was possible to have a digital editorial system that converted a single document to several formats without losing information. The NEJM, in fact, faces that very requirement. If you look at the article in question, you will see that it begins with the article type (“ORIGINAL ARTICLE”), then a headline (“Safety and Efficacy…”), then the authors, then two buckets, one the article, and the other figures/media. The article begins with an Abstract, which begins with Background, Methods, and so forth. The NEJM delivers documents structured in this wise in at least three media: (1) online, (2) PDF, both of which we common readers can read, and (3) in a printed version, which we cannot. (There may also be versions for tablets, library readers, even CD; I don’t know, but you can see the advantage of having a single master document composed according to a formal structure and then converted, as opposed to three or more different documents, all of which must be kept in synch with each other.) Of course, the publisher makes an implicit bargain with the reader that all the versions are identical or, if different, are labeled as such[1]. Or not, as we shall see!

Readers (many of them) like documents that are consistently structured (the formalism is, of course, hidden from them) because structure makes it easy to find what they want. Readers expect to be able to flip to the end of a book to find the index, for example, and so if they encountered a book where the index was in the middle, they would justly conclude either that the publisher’s printer had committed a terrible error, or that the author was some sort of pomo buffoon bent on japery. Readers who value their time, like doctors — I well remember a doctor in my study group, who would exclaim “I could be treating patients!” whenever he felt time was being wasted — appreciate structure even more; if they want to know the methods, they go to the Methods section; if the results, the Results section. If editorial matter does not appear where it is supposed to appear, such readers can and will justly conclude the matter is missing, because that is the bargain the publisher made with them through the pattern and practice of structuring the document consistently, issue after issue after issue.

Having set the scene at tedious length — I really did love that work — we now come to IM Doc, who was — I contend — betrayed as a reader by NEJM’s failure as a publisher to uphold its end of the bargain it made with him when it butchered its document structure and its own editorial standards at a critical juncture[2].

IM Doc, in “An Internal Medicine Doctor and His Peers Read the Pfizer Vaccine Study and See Red Flags [Updated]” wrote:

First, a critical issue for any clinician is “exclusion criteria”…. From my reading of this paper, and the accompanying editorial, one would assume there were no exclusion criteria.

The “exclusion criteria” are to be found through another document element called a Protocol (hat tip Rick in Oregon) about which more in a moment.

They certainly are never mentioned…. And now we know there were exclusion criteria, not because of anything Pfizer, the investigators, or the NEJM did but because of stunning news out of the UK. UPDATE: I will address this at greater length, but an alert reader did find the study protocol, which were not referenced in any way that any of the nine members in my review group could find, nor were they mentioned in the text of paper or editorial, as one would expect for a medication intended for the public at large. I apologize for the oversight, but this information was not easy to find from the article, not mentioned or linked to from the text of the article, the text of the editorial, in the “Figures/Media,” or in a supplemental document.

I need to disentangle this a bit, because I think IM Doc is generously taking on a bit more responsibility than he or needed to.

First, we need to consider what is meant by “the article.” As we know, there are at least three versions of “Safety and Efficacy”: The online version, the PDF version, and the printed version. Which one was IM Doc reading? Certainly the PDF version (assuming the printed version had not yet arrived in the mail[3]). We know that from IM Doc’s text, because he tells us so himself: He cites to “page 5, in Table 1,” to “tables on pages 6 and 7,” and “to tables on page 7.” The online version has no pages. We also know that because of the social setting in which “Safety and Efficacy” was read. Reader KLG described this setting (IM Doc’s “journal group”) in Yves’ follow-up piece as follows:

IM Doc [was part of a] like-minded journal club (a common mechanism for all biomedical scientists and many clinicians to keep up with current developments)

And IM Doc describes the meeting:

we had an ad hoc meeting of our Journal Club to discuss the NEJM article

It is extremely difficult for me to imagine that each member of the Journal Club, whether on Zoom or in person, was reading the online version on a device (one person an Android phone, another on an iPhone, the fourth and fifth on two iPads of different sizes, the sixth on a laptop, and so on). It’s far more likely they were reading the PDF, and most likely printed out, because that way, when one reader says “flip to Table 1 on page 5,” all the readers can easily do that simultaneously (rather than swiping, tapping, scrolling, and so on).

So now we come to the first extremely simple reason why IM Doc and his study group, working from the PDF, could not find the Protocol which contained the exclusion criteria.

They could not find the Protocol element in the PDF because it was not there.

NEJM’s document structure treats protocols as separate, external documents[4], and puts a link to them, called Protocol, within an element called Supplementary Material[5], which is placed at or near the end of the article in the online version only. Here is the online version, which contains the Supplementary Material:

And here is the PDF version, which does not:

Once again, the Supplemental Material, which includes the Protocol with its inclusion criteria, is criticalAs IM Doc writes:

First, a critical issue for any clinician is “exclusion criteria”. This refers in general to groups of subjects that were not allowed into the trial prima facie. Common examples would include over 70, patients on chemotherapy and other immunosuppressed patients, children, diabetics, etc.. This issue is important because I do not want to give my patient this vaccine (available apparently next week) to any patient that is in an excluded group. Those patients really ought to wait until more information is available – FOR THEIR OWN SAFETY. And not to mention, exclusion criteria exist because the subjects in them are usually considered more vulnerable to mayhem than average subjects. From my reading of this paper, and the accompanying editorial, one would assume there were no exclusion criteria. They certainly are never mentioned.

Wnat possible excuse is there for leaving the Supplemental Material out of the PDF? Saving a few bytes?[5]

There is a second, slightly less simple reason why IM Doc and his study group, working from the PDF, could not find the Protocol which contained the exclusion criteria. This concerns how NEJM formats references to the Protocol element in the running text. Here is how references in running text to elements in Supplementary Materials should be formatted, according, at least, to the Journal of the American Medical Associations style guide:

NISO (the National Information Standards Organization) agrees:

So, one would expect references in running text to the Protocol to appear along the lines of “see the Protocol in Supplementary Materials, to flag the “critical part of the evidence” for the reader — that’s one reason we have capitals — and to guide them to its location. That is not what NEJM did. Here is one example:

There’s no formatting in the PDF to indicate what “the protocol” is, and no indication of where it is to be found. All the other mentions of “protocol” are like that. So no wonder — in the trackless wilderness of NEJM’s house style — the Protocol was difficult to find.

 

* * *

 

Summarizing:

(1) NEJM’s editorial practice removes the Supplementary Material element from the PDF version, and hence critical material like the Protocol element that contains the link to the external protocol document, which contains the “exclusion criteria” sought by IM Doc. It is entirely reasonable for readers to expect both online and PDF versions to be identical with respect to critical material.

(2) NEJM’s editorial practice permits the copy desk to format text references to the Protocol element in lowercase, with no indication of the element in which it is to be found. It is entirely reasonable for readers to expect references to critical, named elements to be formatted according to JAMA and NISO standards, wihc at a minimum require initial caps.

(3) The net result of NEJM’s editorial practice was that a study group of nine practicing, busy, and stressed physicians were unable to find critical material affecting treatment because (a) the material was not there to be found and (b) there was no signal to show where it was to be found. NEJM should know its readers better.

(4) NEJM should fix its editorial practice. It’s broken.

NOTES

[1] If matters were otherwise, readers would be put in the impossible and absurd position of having to read all the versions of any given paper and check each for consistency. As it turns out, this is exactly the situation the NEJM has put its readers in. (This reminds me very much of Obama organizers who would say “Go look at the website!” when questioned on details of policy. No.)

[2] The misadventures that came to happen with an article that was one of the most important NEJM published this year, and perhaps for many years, would be a matter for internal inquiry at NEJM. Since the editor-in-chief of NEJM was one of the authors of the “EDITORIAL” that accompanied “Safety and Efficacy,” such an inquiry would presumably be easy to initiate.

[3] The printed version and the PDF version should be identical in any case, because generally the PDF is what the publisher sends to the printer, albeit with the addition of crop marks, etc.

[4] Treating the protocol as an external document makes sense, because the protocol is often quite long, and will be formatted in the style of whoever is submitted the article — in this case, Pfizer — and not in the journal’s house style.

[5] There is another element, confusingly also called Supplemental Appendix and placed at the end of the article. A link to the Supplemental Appendix seems to have made a brief appearance somewhere near the top of the screen for the online version, and then vanished mysteriously, as described by both Yves and reader KLG here. It did not and would not have contained the Protocol element, which goes in the Supplemental Material section.

[6] PDFs can also be made clickable, so not only could the Protocol element have been included within in, it could have linked to the protocol.

 

James Cross

The study states this:

“We assessed the safety and efficacy of two 30-μg doses of BNT162b2, administered intramuscularly 21 days apart, as compared with placebo. Adults 16 years of age or older who were healthy or had stable chronic medical conditions, including but not limited to human immunodeficiency virus (HIV), hepatitis B virus, or hepatitis C virus infection, were eligible for participation in the trial. Key exclusion criteria included a medical history of Covid-19, treatment with immunosuppressive therapy, or diagnosis with an immunocompromising condition”.

Which looks to me like it is clearly stating at least some exclusion criteria: treatment with immunosuppressive therapy or an immunocompromising condition That would make IM Doc’s statement:

“From my reading of this paper, and the accompanying editorial, one would assume there were no exclusion criteria. They certainly are never mentioned’.

Completely wrong.

 
  1. Lambert Strether

    > Key exclusion criteria included

    Come off it. You’re saying that weasel-wording like “included” substitutes for supplying the actual location of the full exclusion criteria.

    Somebody’s completely wrong here, fella, and it’s not me.

     
    1. flora

      +1. Noting the existence of important published material from the same study that’s not included in the main text, but failing to provide a cross reference anywhere in the main text – independent of platform – to the not included important material…. big fail. Scavenger hunts are fine for Halloween Parties, not for medical journals or science journals or tech journals, imo.

       
    2. James Cross

      He wrote “no exclusion criteria” mentioned when “immunosuppressive therapy or an immunocompromising condition” were mentioned.

      And below he asking what to tell his patients on “immunosuppressive therapy” when you can gather even from the original article that it would be untested.

       
      1. IM Doc

        Hence the concern.

        I take care of patients. Many of whom are on all kinds of meds with all kinds of medical problems.

        Many of whom read all about the anaphylactic reactions occurring with this vaccine – and noting that Pfizer, the FDA et al made such a poor show in properly explaining this to the public that they are looking to their family or primary care doctors to help explain this to them. Many of whom are wondering if the same thing that befell the anaphylaxis patients will befall them because of XYZ med or medical condition. It is a truly rational question, and is coming at me all the time because of the extreme curiosity about all things related to COVID vaccine in this emergency.

        Again, unlike many other research trials I have ever looked through, this one does a very poor job of explaining whether immunosuppressive therapy is an exclusion criteria because of possible issues that occurred in Phase I trials – or if they just cut it out all together. This is true of many other medical problems on that list. I am sorry that you are unable to see how the former would be much more problematic for safety concerns in a patient’s mind – and the latter not so much. But again, it has fallen to me and thousands of other PCPs like me all over this country and world to discuss this with patients. It is happening all day every day. And Pfizer/FDA have given us very limited tools to help people understand. It is causing significant hesitation in a lot of patients, even health care workers (https://www.beckershospitalreview.com/workforce/staff-at-hospitals-in-dc-texas-turn-down-covid-19-vaccine.html). If the goal was to vaccinate everyone, they have sure started us off with a very troubled gameplan. This is just one example.

         
  2. IM Doc

    I think this point needs to be made. Once eventually found, the exclusion criteria were reported in the protocol in a very incoherent manner.. As to your point above, you can easily see that there were way more exclusion criteria than just the simple things mentioned above in your comment. And because I have been reading these things for decades, I knew that was going to be the case.

    Again, once found, here is the mess of the manner they were presented – this is a word for word cut from the original document:

    •Other medical or psychiatric condition including recent (within the past year) or active suicidal ideation/behavior or laboratory abnormality that may increase the risk of study participation or, in the investigator’s judgment, make the participant inappropriate for the study.
    •Phases 1 and 2 only: Known infection with human immunodeficiency virus (HIV), hepatitis C virus (HCV), or hepatitis B virus (HBV).
    •History of severe adverse reaction associated with a vaccine and/or severe allergic reaction (eg, anaphylaxis) to any component of the study intervention(s).
    •Receipt of medications intended to prevent COVID 19.
    •Previous clinical (based on COVID-19 symptoms/signs alone, if a SARS-CoV-2 NAAT result was not available) or microbiological (based on COVID-19 symptoms/signs and a positive SARS-CoV-2 NAAT result) diagnosis of COVID 19.
    •Phase 1 only: Individuals at high risk for severe COVID-19, including those with any of the following risk factors: ◦Hypertension
    ◦Diabetes mellitus
    ◦Chronic pulmonary disease
    ◦Asthma
    ◦Current vaping or smoking
    ◦History of chronic smoking within the prior year
    ◦BMI >30 kg/m2
    ◦Anticipating the need for immunosuppressive treatment within the next 6 months

    •Phase 1 only: Individuals currently working in occupations with high risk of exposure to SARS-CoV-2 (eg, healthcare worker, emergency response personnel).
    •Immunocompromised individuals with known or suspected immunodeficiency, as determined by history and/or laboratory/physical examination.
    •Phase 1 only: Individuals with a history of autoimmune disease or an active autoimmune disease requiring therapeutic intervention.
    •Bleeding diathesis or condition associated with prolonged bleeding that would, in the opinion of the investigator, contraindicate intramuscular injection.
    •Women who are pregnant or breastfeeding.
    •Previous vaccination with any coronavirus vaccine.
    •Individuals who receive treatment with immunosuppressive therapy, including cytotoxic agents or systemic corticosteroids, eg, for cancer or an autoimmune disease, or planned receipt throughout the study.
    •Phase 1 only: Regular receipt of inhaled/nebulized corticosteroids.
    •Receipt of blood/plasma products or immunoglobulin, from 60 days before study intervention administration or planned receipt throughout the study.
    •Participation in other studies involving study intervention within 28 days prior to study entry and/or during study participation.
    •Previous participation in other studies involving study intervention containing lipid nanoparticles.
    •Phase 1 only: Positive serological test for SARS-CoV-2 IgM and/or IgG antibodies at the screening visit.
    •Phase 1 only: Any screening hematology and/or blood chemistry laboratory value that meets the definition of a ≥ Grade 1 abnormality.
    •Phase 1 only: Positive test for HIV, hepatitis B surface antigen (HBsAg), hepatitis B core antibodies (HBc Abs), or hepatitis C virus antibodies (HCV Abs) at the screening visit.
    •Phase 1 only: SARS-CoV-2 NAAT-positive nasal swab within 24 hours before receipt of study intervention.
    •Investigator site staff or Pfizer employees directly involved in the conduct of the study, site staff otherwise supervised by the investigator, and their respective family members.

    I have never seen such a mess in my entire life as a physician – and on such a critically important issue. I have never seen haphazard dumping of Phase 1 exclusion together with Phase 3. It is one thing to do this to a boutique oncology drug – it is in another realm to do this to something we are going to be giving to everyone.

    Please tell me – what do I tell my patients on immunologic agents like ENBREL about their safety profile for this vaccine? It has already come up twice this week. It is very clear they were excluded from phase 3 – but WHY? – did something adverse happen to these people in PHASE 1 or PHASE 2 – it certainly isn’t clearly mentioned. Did the investigators just cut them out from the start – again – does not seem to be very clear. These types of issues are usually clearly laid out in a much more functional table than this mess. What about pregnant patients? Did something concerning happen in the PHASE 1 trials – I may have overlooked in this mess – but I cannot see they were excluded from Phase 1 trials? Usually, they would be, and a simple note about off the bat exclusion would have been perfect. But really not obvious in this mess. Again, not totally clear exactly why they were excluded.

    I have mentioned this before, but again, there is a part of me really wanting to know why exclusion criteria were even allowed in a study with an emergency drug meant for every human being on earth. It just does not make a lot of sense.

     
    1. James Cross

      So you don’t like how they wrote it?

      Most of those exclusions seem actually to be for Phase 1 which will have nothing whatsoever to do with what you tell your patients. Phase 1 exclusions would always be very extensive because it is mainly trying to get a rough idea of the safety of the vaccine on a very small number of people before trying it on larger groups. So the objective is to not throw a bunch of complications into a determination of basic safety on a small group. It is also to determine dosage ranges that will NOT cause severe side effects. As a doctor you should know this. I would be surprised if Phase 1 had much more than a 100 volunteers.

      I know more about the Moderna vaccine and it included 250 micrograms doses in Phase 1 and had severe reactions from several people. It also determined that the 100 microgram dose produced sufficient antibodies without severe reactions so the other phases used 100 microgram doses.

       
      1. IM Doc

        Please answer the question –

        What do I tell my patients on Enbrel this week about their safety fears with this vaccine? – And I would add that one of them is a retired chairman of a department in an Ivy League medical school who herself could not figure this out on her own. She had seen her rheumatologist on Zoom the day before me. They could not figure it out together. That is why she brought it up with me. Look at the above presentation and tell me.

        And, no, I do not like how they wrote it. That is the entire point. They are not all just Phase 1 exclusions. They are indeed written in a very opaque way. They are all thrown together instead of neatly separated. I have 20 patients a day when we are not having COVID on top of that with 6-10 in the hospital. I am profoundly busy right now. I long for the days when information like this was presented where a busy clinician could advise his patient with confidence without having to do a deep dive in data sets for minutes to hours (assuming it could be found easily) and having to make suppositions on top of that. Again, my patient, a retired dept chair in medicine, and her rheumatologist could not figure this out either.

        I think this slide to sloppiness has been going on for some time. I have seen it slowly happening. It is just now becoming apparent to many of us with a degree of urgency because we as clinicians and our patients themselves are looking at this material carefully and much more than usual because of this emergency.

         
        1.  
  1. James Cross

    The study states this:

    “We assessed the safety and efficacy of two 30-μg doses of BNT162b2, administered intramuscularly 21 days apart, as compared with placebo. Adults 16 years of age or older who were healthy or had stable chronic medical conditions, including but not limited to human immunodeficiency virus (HIV), hepatitis B virus, or hepatitis C virus infection, were eligible for participation in the trial. Key exclusion criteria included a medical history of Covid-19, treatment with immunosuppressive therapy, or diagnosis with an immunocompromising condition”.

    Which looks to me like it is clearly stating at least some exclusion criteria: treatment with immunosuppressive therapy or an immunocompromising condition That would make IM Doc’s statement:

    “From my reading of this paper, and the accompanying editorial, one would assume there were no exclusion criteria. They certainly are never mentioned’.

    Completely wrong.

     
    1. Lambert Strether

      > Key exclusion criteria included

      Come off it. You’re saying that weasel-wording like “included” substitutes for supplying the actual location of the full exclusion criteria.

      Somebody’s completely wrong here, fella, and it’s not me.

       
      1. flora

        +1. Noting the existence of important published material from the same study that’s not included in the main text, but failing to provide a cross reference anywhere in the main text – independent of platform – to the not included important material…. big fail. Scavenger hunts are fine for Halloween Parties, not for medical journals or science journals or tech journals, imo.

         
      2. James Cross

        He wrote “no exclusion criteria” mentioned when “immunosuppressive therapy or an immunocompromising condition” were mentioned.

        And below he asking what to tell his patients on “immunosuppressive therapy” when you can gather even from the original article that it would be untested.

         
        1. IM Doc

          Hence the concern.

          I take care of patients. Many of whom are on all kinds of meds with all kinds of medical problems.

          Many of whom read all about the anaphylactic reactions occurring with this vaccine – and noting that Pfizer, the FDA et al made such a poor show in properly explaining this to the public that they are looking to their family or primary care doctors to help explain this to them. Many of whom are wondering if the same thing that befell the anaphylaxis patients will befall them because of XYZ med or medical condition. It is a truly rational question, and is coming at me all the time because of the extreme curiosity about all things related to COVID vaccine in this emergency.

          Again, unlike many other research trials I have ever looked through, this one does a very poor job of explaining whether immunosuppressive therapy is an exclusion criteria because of possible issues that occurred in Phase I trials – or if they just cut it out all together. This is true of many other medical problems on that list. I am sorry that you are unable to see how the former would be much more problematic for safety concerns in a patient’s mind – and the latter not so much. But again, it has fallen to me and thousands of other PCPs like me all over this country and world to discuss this with patients. It is happening all day every day. And Pfizer/FDA have given us very limited tools to help people understand. It is causing significant hesitation in a lot of patients, even health care workers (https://www.beckershospitalreview.com/workforce/staff-at-hospitals-in-dc-texas-turn-down-covid-19-vaccine.html). If the goal was to vaccinate everyone, they have sure started us off with a very troubled gameplan. This is just one example.

           

Profile

temporaryreality: (Default)
temporaryreality

June 2024

S M T W T F S
       1
2345678
9101112131415
16171819202122
23242526272829
30      

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Dec. 30th, 2025 02:57 am
Powered by Dreamwidth Studios