Improving feedback from the MRCGP CSA examination

Improving feedback from the MRCGP CSA examination

Dr Mahibur Rahman

We are often contacted by GP registrars or GP trainers requesting support with understanding the feedback from the MRCGP CSA. Many doctors have commented that they find the feedback difficult to interpret. This has been recognised as an important issue and recently a motion was passed at the LMCs conference calling for immediate improvement in the feedback from the CSA. In this article Dr Mahibur Rahman looks at the current feedback, the areas that could be improved and suggestions on ways to make the feedback clearer and more helpful for both trainers and registrars.

Understanding the current CSA feedback

Currently there are 2 main sections to the feedback from the CSA. The top part gives the candidate’s total score from all 13 cases (out of 117), with the pass mark for the date they sat the exam. This total score is based on the summative part of the assessment, which is based on 3 domains for every case: data gathering, clinical management, and interpersonal skills.

For each domain, a candidate is graded with a score attached to each grade as follows: Clear pass: 3 marks, Pass: 2 marks, Fail: 1 mark, Clear fail: 0 marks. This gives a total score for each case of between 0 and 9.

To gain a pass, a candidate must get an overall score equal to or above the pass mark for a given day. This is adjusted each day using the borderline group method to ensure the standard of the exam remains the same each day. The actual pass mark is variable with a usual range between 72 and 77 out of 117.

The second part of the feedback is formative – it relates to the 16 feedback statements provided by the RCGP in a grid. This grid can provide information on consulting areas that a candidate could improve on. It is important to understand that this part does NOT determine the score or whether a candidate has passed or failed – it is formative, and aimed at helping doctors identify areas of their consulting that they could improve. The current feedback looks like this:

CSA feedback current

What are the problems with the current feedback?

There is no breakdown of the marks awarded from each case (out of 9), and no way for a candidate or trainer to see clearly if marks were dropped in data gathering, clinical management or interpersonal skills for each case, or as a general trend over the course of the whole exam.

In some cases, the formative feedback can help identify areas to work on, but in some cases it can lead to confusion. A common source of confusion relates to the fact that candidates with the same number of crosses can have very different scores. Finally, where a candidate has no crosses relating to a specific case, many candidates think that it means they must have scored very well, or at least gained 6 or more marks out of 9. However it is impossible to tell how well or poorly they have performed in that case from the lack of crosses– they could have scored anywhere from 0 to 9. This is because:

  • The formative feedback does NOT determine the score for a case – this is determined by the performance in the 3 domains being assessed. Scores for these are not provided in the current feedback as standard – candidates that want to access these scores can request their mark sheets under the Data Protection Act.
  • Only feedback statements that were flagged in 2 different cases show up in the feedback provided to candidates – there are hidden crosses where a statement was only flagged in a single case. A candidate with no crosses could actually have had several crosses relating to feedback statements that did not occur again in other cases. This could have led them to score very poorly in that case, but they would not know it from looking at the feedback.

This candidate failed the CSA by a few marks – look at the formative feedback for their first 3 cases:

CSA formative feedback - current

This candidate scored 7/9 for the first case (joint problems), and 2/9 for the second case (acute illness), but there would be no way to know that they had performed really poorly in the second case from the current feedback. There were actually 3 feedback statements that were flagged in this case, but they don’t show up because those statements did not apply to any other case (and currently these statements are hidden).

How could the feedback be improved?

The GPC motion called for “the feedback from the MRCGP exams to be improved immediately”. Here are 3 simple ways that the feedback could be made clearer and more effective in helping identify areas to work on to improve performance. They can all be introduced using data that is already collected in the exam, and so could be implemented quickly with little additional cost.

1. Provide a breakdown of total marks for each domain as well as the total score. In the AKT, candidates get a breakdown of their scores in the 3 domains (clinical medicine, organisational, and evidence interpretation). This will give a clearer indication of any weaker areas overall:

New CSA feedback - summative

This candidate and their trainer can immediately see that they could make improvements in all parts of the consultation, but that the clinical management domain was their weakest overall. This may allow more targeted work on this part of the consultation. Without this information, this candidate (and their trainer) may focus more on the interpersonal domain, without realising that although this could be improved further, this is actually their strongest domain overall.

2. Provide the domain scores for every case as well as the formative feedback. Taking both the summative and formative feedback together provides more meaningful information and will allow easier identification of both consulting skills and curriculum areas that need improving. This could be provided by adding a separate table for the domain scores:

CSA domain feedback for individual cases

Looking at this, it is clear that this candidate had 2 cases where they performed very poorly – the young adult female with an acute illness, and the middle aged female with a women’s health issue. These may be areas that they struggle with, and indetifying them will allow focused improvement in knowledge.

3. Provide details in the formative feedback section of ALL statements that were flagged, even when this only applied to a single case. This will allow candidates to identify all areas that examiners felt they could work on – even candidates that have done well can benefit from knowing areas that they could improve. Combined with the summative feedback above, this would also make it easier to separate a candidate that is below the pass standard in multiple areas of multiple cases from one that had a couple of really poor cases due to poor knowledge of a specific curriculum area, or because they missed something key in that case. Here is the formative feedback from those first 3 cases that we looked at earlier; the second image shows all crosses (those that were previously hidden are shown in red for clarity):

Current feedback:

CSA formative feedback - current

Proposed feedback:

Proposed feedback showing all crosses

You can see that taking this with the domain scores, it is immediately clear why this candidate got such a low score in the acute illness case, and that had they performed better in this case, they may have passed. This would also help candidates understand their performance better. From the current feedback they may think that this was one of their better cases when actually it is their worst. Providing this extra information does not give any information that will jeopardize case security, but it does provide more meaningful information for someone trying to improve.

How it would look together

All the feedback would fit onto 1 A4 page, allowing quick cross referencing between the different sections. This is how the new feedback could look in the e-portfolio.

New CSA feedback - summative

CSA domain feedback for individual cases

New CSA feedback

Summary

It is clear that further research needs to be carried out to investigate the possible reasons behind the differential pass rates in different groups – however this will take time. By improving feedback immediately, we can ensure that candidates and trainers have clearer, more effective feedback. All these changes can me made using data that is already being collected, so this could be implemented quickly and with little additional cost. Hopefully this will enable more focused work on the key consultation skills that an individual doctor may need to work on to help them improve and pass the exam.

Are you a GP trainer or a GP registrar? What do you think about these ideas for improving the feedback from the CSA? Please share your thoughts!

Advertisements

MRCGP CSA Exam Feedback and Summary – February 2012 exams

MRCGP CSA Exam Feedback and Summary – February 2012 exams

Dr Mahibur Rahman

After each MRCGP CSA examination, the examiners release a report highlighting key information from the last exam. This includes pass marks and rates for the sitting, along with the number of candidates sitting the exam. Since the February 2012 exam they also started releasing a feedback report highlighting key areas that candidates found challenging.

These topics are likely to continue to feature in future CSA sittings, as there is a common case bank, so it is worth ensuring that you have a good understanding of how to tackle them.

If you are thinking of sitting the MRCGP CSA in November 2012 or January / February 2013, then you have probably started preparing. As the January / February sitting is the most popular each year, we thought it would be helpful to look at the feedback and challenging areas from this sitting in 2012. The sumary report for the May 2012 CSA exam is not yet available.

Key facts from the February 2012 MRCGP CSA exam:

Number of candidates: 2074

Proportion sitting the CSA for the first time: 92.5%

Overall pass rate: 71.8% (1490 candidates passed, 584 candidates failed)

The top score was 111 out of 117
The mean score was 81 out of 117
The lowest score was 37 out of 117
97 candidates (4.7%) scored 100 or more out of 117
67 candidates (3.2%) scored 20 or more marks below the pass mark.

Challenging areas

The examiners’ report from the February 2012 diet of the MRCGP CSA exam was released in April, and highlighted the following areas that caused candidates difficulty:

Genetics in primary care

Cases involving genetics regularly cause CSA candidates problems in the exam. Examples of cases you should be prepared to handle include:

Prenatal counselling for risk of single gene disorders – e.g. sickle cell disease, Huntington’s, neurofibromatosis, cystic fibrosis etc.

An asymptomatic patient requesting a colonoscopy with a family history of colon cancer.

While you do not need to have an in depth knowledge of specific genetic disorders, you should be able to take a good history and draw a family tree. You should also be able to explain the difference in risk for autosomal dominant and autosomal recessive disorders, and know when it is appropriate to refer to a genetics counselling service.

Examinations

In some cases in the CSA you will actually perform a physical examination. In some cases, candidates lost marks for being unable to be focused in their choice of examination, or not being able to perform the examination proficiently. Examples of a lack of focus would include requesting a full physical examination in someone with hearing loss – it would be more appropriate to examine the ears, and to perform a Rinne and Weber test. Examples of an inadequate examination highlighted by the examiners included listening to a patient’s chest with through their shirt! Most examinations in the CSA are fairly straightforward – you should try to practice all the common examinations with a study group until you are fluent. Ask your trainer to observe you and to provide feedback.

The MRCGP CSA is a challenging, comprehensive examination, so it is important that you start preparing for it early. Try to get as many observed consultations as possible with your trainer, and form a study group early on.

Further reading:
Complete February 2012 CSA Summary report

The Emedica MRCGP CSA Course includes teaching on the new CSA mark scheme including the new 2012 CSA feedback statements. Each course only takes 6 GP registrars, with a strong emphasis on practice with individual feedback. Practice sessions are donw in groups of 3, allowing each candidate to have 4 mock CSA practice cases. There is detailed, constructive 1 to 1 feedback after each case using the new marking criteria. Our mock CSA cases are done in a realistic setting with professional simulated patients and timed in the same way as the real exam. Alumni members can get a £20 discount by using the code alumnimrcgp

 

MRCGP CSA Preparation – Tips to help you pass the MRCGP CSA exam

CSA Preparation – Tips to help you pass the MRCGP CSA exam
Dr. Safiya Virji

The MRCGP CSA examination is a challenging exam. Dr Safiya Virji sat and passed the exam on her first attempt with one of the top scores in the country. In this article she shares some tips on preparing for the CSA exam.

An important step in successfully passing the CSA is to make a decision early on which sitting to go for, and actively work towards being thoroughly prepared by this time. I made the decision six months beforehand. I had just started my ST2 placement in GP and took this opportunity to practice various consultation styles until I found one that suited me and came more naturally to me.

I ensured that I was videoed frequently from very early on and did not let the embarrassment of watching myself on the screen stop me from getting vital feedback from my trainer. I was always conscious to encourage my colleagues to give constructive criticism so that I had identified specific points to improve over the coming weeks. I also gradually reduced my consultation times; starting at 20 minutes and gradually working down to 10 minutes about two months before the exam.

I found joint surgeries with my trainer and other partners at the surgery extremely beneficial. Not only can you see alternative ways of phrasing things, but it also gives you a chance to see how you are inclined to perform when you don’t know who or what is going to come through the door and you are being watched. Conducting these on a weekly basis meant by the time the exam came, I was already comfortable with the scenario of being watched conducting consultations I was not familiar with, so was less nervous then you would expect in such a weighty exam. This enabled me to perform to the best of my abilities when it really counted.

It is important to use all opportunities as exam practice. Every patient you see is an opportunity to practice explaining a diagnosis in layman’s terms or to ask about what they think may be causing their presenting complaint. Anything that goes wrong in your consultations is worth jotting down and discussing with your trainer afterwards. It could be the phrasing, or it could be bad choice of questions. I found the more that went wrong, the better I was getting, as I was prepared for all eventualities.

I was also part of the on-call rota and working on emergency patients with a seven minute consultation time from about four months before the exam. This in combination with the GP out of hours (OOH) gave an excellent opportunity to practice for the exam. I took the opportunity in OOH to ensure every patient I saw was observed by the trainer, timed and feedback was given.

I read a selection of case books including:
nMRCGP – Practice Cases, Clinical Skills Assesment – Raj Thakkar
CSA Scenarios – Thomas M Das
Consultantion Skills for the new MRCGP – P Naidoo & C. Monkley
Get Through New MRCGP: Clinical Skills Assessment – Bruno Rushforth (this was the best book for role play in my opinion)

I tried to use these for role plays with colleagues at least for a one hour session a week, increasing this to 2-3 hours a week in the last month before the exam. However, I also read these books in my own time to learn how simple presenting complaints can often have an unbelievable amount of depth, and how missing out one vital question can lead to missing the underlying point of that consultation.

Regarding examinations, I watched videos on how to do the examinations thoroughly yet quickly, and used my own equipment to perform these multiple times on family and friends in the week before the exam so examinations were fresh to hand if necessary.

In preparation for the exam, I set myself a target to complete all consultations within 8 minutes. Strict time management a month before the exam meant I was used to working at a fast enough pace that would ensure I would not over run in the exam, even with the unexpected cases which take a few seconds longer.
I ensured I was always trying to examine the patient by 5mins, and always kept a note of the start and finish time of each consultation to ensure I was always working to time.
In the exam, on several occasions I was still conversing with the patient when the bell rang. However, it seemed I covered enough material at the end of the 10 mins to pass well on each station.

Having sat the exam before all my colleagues, I was not exposed to many horror stories about the exam. This meant I was optimistic when exam time came. A combination of this, alongside minimal nerves and trying my best to apply a structure that worked for me when things were going to plan, and adapting my consulting style when I needed to be flexible, plus a happy face, lead to my passing with a score of 106/117. And I forgot to mention, when I sat the exam I was 38weeks pregnant!

One month before the exam I went on several courses, one of which was the Emedica MRCGP CSA course. I found the course useful as it tackled preparation slightly differently to other courses. The group was very small (courses take just 6 candidates per day). This meant there was time to focus on each trainee independently and specific feedback was given on their performance. By the time you go on any course, you have usually had experience of the simple well known cases, but the scenarios at the Emedica course were slightly more complex then average which meant you were prepared for the more challenging cases in the real exam.

Some of the cases in the CSA do throw you so having some practice at performing under pressure can mean the difference between a pass and fail. The feedback on the course was very useful as it was not based just around what was done well, but more on what needed to be improved on in order for you to pass, and pass well. For me, this approach was more beneficial as I always maintained the attitude that by taking constructive criticism on board, I was far more likely to pass as all my flaws would be ironed out by the time the exam came round. After the course, further reading material and links were provided, including videos of common examinations. This was very helpful as though it is not a huge part of the exam, when it does come up, it is essential they are performed fluently and effectively so that the correct diagnosis is made.

In summary, the my key tips to help you pass the MRCGP CSA exam are:

• Give yourself enough time to prepare – I started 6 months before the exam
• Use all opportunities as CSA practice opportunities – surgery, OOH, on call
• Be observed as much as possible – joint surgeries and video surgeries are both helpful
• Actively seek constructive feedback – and use it to develop your technique
• Create an effective structure that works for you and apply it as much as possible
• Get used to working under time pressure – being comfortable with 10 minute consultations really helps
• Don’t let stress on the day change your attitude toward exam consultations – keep calm and carry on

One last point, once you have finished with one patient, don’t analyse or get upset in the exam, move your focus onto the next one and give it your best!

Dr Virji is a GP Registrar (ST3) in Oxford Deanery. She passed the MRCGP CSA exam on her first attempt, and scored 106 marks out of a maximum of 117.

The Emedica MRCGP CSA Course includes teaching on the new CSA mark scheme including the 2012 CSA feedback statements. Each course only takes 6 GP registrars, with a strong emphasis on practice with individual feedback. Practice sessions are donw in groups of 3, allowing each candidate to have 4 mock CSA practice cases. There is detailed, constructive 1 to 1 feedback after each case using the new marking criteria. Our mock CSA cases are done in a realistic setting with professional simulated patients and timed in the same way as the real exam.

How to pass the MRCGP CSA: Understanding the new MRCGP CSA Mark Scheme

Pass the MRCGP CSA: Understanding the new MRCGP CSA Mark Scheme

Dr Mahibur Rahman.

You may have heard that the pass rate for the MRCGP CSA dropped recently, with only 46% of candidates passing the September 2010 sitting of the exam (the pass rate was around 80% previously). Many trainees have suggested that the high CSA failure rate is due to the new marking scheme.

The RCGP changed the way the MRCGP CSA exam is marked from September 2010. We posted an update about this a while back, but wanted to go through the details of the new marking scheme in more detail.

With the previous method of marking, candidates received an overall mark for each case, and had to pass 8 out of 12 cases to get an overall pass in the CSA. With the new mark scheme, candidates do not “pass” or “fail” each case, but are instead given a numerical score for each of 3 domains in every case. The 3 domains are Data Gathering, Clinical Management, and Interpersonal Skills. The numerical scores are:

Clear Pass – 3
Pass – 2
Fail – 1
Clear Fail – 0

This gives a maximum score for each of the 13 cases of 9 (a clear pass in each of the 3 domains). The total score for each candidate is then calculated by adding up the scores from each case, and is out of a maximum of 117 (9 x 13 cases). This allows you to compensate for a poor performance in one case with a very good performance in another case.

The pass mark each day is set using the borderline group method, which allows for adjustment depending on the difficulty of cases on the day. In September, the pass mark ranged from 75/117 to 77/117. A candidate that had a pass in every domain in every case would score 78/117 and so would have passed the September sitting.

It is not possible to directly compare the two mark schemes without more data than is currently available on the individual domain scores for candidates in the old scheme. However, it does seem that with the new marking scheme, passing the CSA is significantly more difficult than with the old marking scheme, where a candidate could have had 4 clear fails and 8 marginal passes and still achieved an overall pass.

Some details of the new mark scheme were available on the RCGP website, but it seems that many trainees were not aware of the changes before sitting their CSA in September. The RCGP has published some further details about the new marking scheme, including answers to some frequently asked questions.

Given the more challenging CSA mark scheme, we recommend that trainees start practising for the exam earlier on. Some of the ways you can improve your technique are:

• Understand what the exam is testing – read through the RCGP CSA feedback statements and examiners suggestions on how to improve on each one – many trainees only read this if they fail the CSA and are preparing for a resit. If you can learn what makes people fail, you will know what to avoid.
• Set up a study group with other trainees and try to practice cases regularly – perhaps once a week from the end of your ST2 year
• Try to do joint surgeries with your trainer so you can get feedback on both the communication and clinical aspects of your case.
• Try to do some video surgeries – you will need to enlist the help of your practice team for this to work effectively. Ask the receptionists to let your patients know that you are videoing as part of your training, and ask for a consent form to be signed if they are willing to take part. Make it very clear to patients that these videos will NEVER be used for anything other than your training, and that they can change their mind at any time during the consultation. Watching yourself on video, or going through them with your trainer often helps to pick up communication issues that could otherwise be missed.

The CSA was always a challenging assessment, with the new mark scheme it is important that trainees get as much practice as possible, with honest, constructive feedback on their performanceearly on, to allow time to embed any changes before the exam.

The Emedica MRCGP CSA Course includes teaching on the new mark scheme. Each course only takes 8 GP registrars, allowing each candidate to have 3 mock CSA practice cases. There is detailed 1 to 1 feedback after each case using the new marking criteria. Our mock CSA cases are done in a realistic setting with professional simulated patients and timed in the same way as the real MRCGP CSA exam.

Changes to the MRCGP CSA Exam

The RCGP has announced important changes to the number of cases and marking of the MRCGP CSA. From the September 2011 sitting of the exam, all 13 cases will count compared to the previous 12 cases + 1 pilot case. The way that the overall marks for the exam and the way a Pass or Fail for the overall exam is decided will also change.

In previous sittings, examiners marked each candidate in 3 domains, and then awarded one final grade based on their overall impression of the performance for that case. The grades were – Clear Pass, Marginal Pass, Marginal Fail, and Clear Fail. Only the grade for the overall impression for each case counted towards your exam result. To get an overall pass in the CSA, candidates needed to get a pass in 8 or more out of the 12 assessed cases.

From September, the examiners will not give a separate grade based on the overall performance. Instead, the grades given for the 3 domains (Data gathering, Interpersonal Skills and Clinical Management) will be converted to a numerical value, with the total score for each case being the total score from each domain.

The pass mark will no longer be fixed, but will instead be set each day using the borderline group method, which has previously been used to set the pass standard for the PLAB OSCE examinations.

The Emedica CSA Preparation Course includes teaching on the new mark scheme. Each course only takes 6 GP registrars, allowing each candidate to have 4 mock CSA practice cases. There is detailed 1 to 1 feedback after each case using the new marking criteria. Our mock CSA cases are done in a realistic setting with professional simulated patients and timed in the same way as the real exam.

Further reading:
RCGP information on the changes to the CSA.