Air France is a global airline carrier with a mobile problem—their mobile app experience is terrible. As of May 2019, their app has a rating of 2.5 (on the iOS store) with users complaining about it being the “worst app ever experienced” which has “error messages all the time [which] provides user no cause of errors or how to correct them”. In order to see if we could address some of the real-user concerns that were not related to technical (i.e. development) issues , my partner and I decided to conduct a heuristics evaluation and redesign on what we believed to be the core offering of the app, the ‘Purchase a Ticket’ experience.
There were three main areas of focus that we established a redesign would be the most beneficial in order to improve the user experience, specifically: the hierarchy of information, the composition of the elements, and the helpfulness of error messages.
As this was an educational class-project, there has not been any real-world impact although we did think it would be interesting to release the designs to Air France to see if they would use them in their actual application.
The main takeaway for me was that the usability evaluation portion could have been made less subjective between my partner and I if we created a more precise custom-scoring system (or just used the Neilen Norman scoring system) in addition to having guiding prompts when evaluating each heuristic. The increased subjectivity meant that our scores (out of 5) for each heuristic often resulted in us defending a particular score rather than coming to the same "conclusion".
In order to see our process, please continue reading further below.
In this partner project, we selected the Air France mobile app as it had a rating of 2.5 on the iOS app store. Countless reviews complained about “error messages everywhere” and that the messages provided users with “no cause of errors or how to correct them”.
We also downloaded the app to informally test it for ourselves, and found that there was much opportunity to improve the experience due many factors including poor user interface design, poor microcopy, and unexpected navigational interactions.
There were two main goals for this project:
The reason a heuristic evaluation is important to conduct is because it exposes usability issues that a user may encounter, according to certain rules of thumb. It is typically conducted by at least three usability experts, and while there are more than 200 criteria which can be used to conduct an evaluation, many refer to the 10 Usability Heuristics for User Interface Design by Jakob Neilsen.
For our heuristic evaluation, we focused on seven of the 10 heuristics by J. Neilsen. These were:
Instead of using Severity Ratings for Usability Testing by Jakob Neilsen, we devised our own rating system because we felt it yielded greater flexibility (see Challenges at end of case-study). Each heuristic was given a score out of five using the rubric shown below:
To conduct the heuristic evaluation, our process involved each of us going through the task flow with each heuristic in mind , and then assessing a score. We then shared our scores for each heuristic, and if we did not align in the score, we then discussed and defended our position by explaining why the score we gave was "correct" .
The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.
A processing system modal was present whenever there was a delay or between page transitions, which is helpful so the user doesn't think the system has frozen.
The title Purchase a Ticket is the same on all screens after tapping on Search, which is confusing for the user as to their exact location in the flow.
We recommend having more specific page titles on each screen to allow for easier localization for the user, e.g. Departure Ticket, Return Ticket, Add Options, Personal Details.
Users often make mistakes and need 'emergency exits' to leave the unwanted state.
The user is able to exit from the Purchasing a Ticket flow, which takes them back to Buy Now.
The back button on each screen does not behave consistently. In some instances it takes user back to Buy Now (the first screen of the flow), and in others, it takes them back to the previous screen.
When a user taps on the processing system modal before the screen loads, it will leave them viewing a blank screen. There is no indication of what is going on, which is confusing for the user.
Users shouldn't have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.
Icon usage for additional flight amenities are consistent with what we would expect.
There is incorrect usage of the system error colour for the information modal options. Additionally, the positive action is placed on the left.
The combination of the magnifying glass icon and Buy Now label is confusing as the icon typically indicates searching behaviour, but that word is not present in the label.
Prevent problems from occurring in the first place or check for them and present users with a confirmation option before they commit.
There is no dynamic error prevention on any of the relevant form fields such as phone, or e-mail.
We recommend that dynamic field validation should be provided on appropriate form-fields because it is frustrating for the user when they attempt to transmit their information, but instead receive an error notification and have to go back to the offending fields.
Minimize memory load by making objects, actions, and options visible. Instructions should be visible or easily retrievable.
Buy Now screen shows previously searched flights, so that the same information does not have to be input again if the user is interested in looking at that particular trip again.
Form fields do not display previously entered information when a revisiting user.
We recommend that the system should save and suggest previously entered information for ease-of-use for the user.
Dialogue should not contain information which is irrelevant or rarely needed.
The overall design of screens is overcrowded, and it is not easy to discern what content the user should be paying attention to. There is poor use of negative space between sections and elements, poor use of colours, and poor typographical hierarchy.
The Youth/Senior Ticket information is present at the bottom of all the ticketing screens. This means that users it applies to may miss it, and that users who it does not relate to may unnecessarily see it.
Error messages should be expressed in plain language, indicate the problem, and suggest a solution.
The system provides a useful error message if no flight is available for specific travel dates.
The error modal shown on the personal details page is extremely hard to read as it is poorly formatted, and does not suggest how the user can rectify them.
We recommend providing a better-organized error modal to reduce the cognitive load on the user when they need to rectify their mistakes.
Our heuristic evaluation provided the purchase a ticket flow with a total score of 21/35. While our evaluation revealed that there were no disastrous usability issues for a user when attempting to purchase a ticket, there was a lot of room for improvement to make the flow more usable, and a better overall experience.
The redesigned screens were based off of the recommendations that we had made during the Heuristic Evaluation. Going into the redesign, we anticipated that the largest visual overhaul would be required on the Select Departure Ticket; Trip Options would also require some effort to redesign visually; the Personal Information screen would look very similar, but have usability and visual improvements.
In the previous experience (left screen for each pair), all screens were titled Purchase a Ticket which made it confusing for the visitor to orient themselves in the ticket purchase flow. In the redesigned experience, we changed the title to be more descriptive, such as Select Departure Ticket, Trip Options, and Personal Information, to improve the experience.
In the previous experience, tapping Tap to Cancel (left screen) would stop all requests and leave users waiting for something to happen on an blank screen if they cancelled before content had loaded. To improve the experience, tapping the modal should always take the user back to the previous screen by preventing a "dead" page. Notice the microcopy change too.
Another issue regarding this heuristic was that in the previous experience, tapping the back chevron in the navbar would behave inconsistently. Sometimes it would take the user all the way back to Buy Now, and othertimes it would take the user back to the previous screen. To improve the experience, tapping the back button should always take the user back to the previous screen.
In the previous experience, the positive action was placed on the left, and the system error colour was used in an incorrect manner to highlight it. To improve the experience, the positive action is placed on the right, and the font colour is changed to dark blue as it is not an error.
In the previous experience, the Buy Now label paired with the magnifying glass icon to represent searching for tickets could be made more clear. To improve the experience, we changed the label say Search Flights instead, which is more a more accurate indication of what the user will be doing.
In the previous experience, there was no dynamic error validation when users entered their information into the phone number or e-mail address form fields. The only time the user would discover if there was an error was after they attempted to submit their information, after which they would receive a notification that there was a mistake and then have to scroll to those fields to correct them. To improve this experience, the user is informed if their entry is valid/invalid a short time after they have stopped entering information into that field so that they don't have to experience the hassle of the previous experience.
In the previous experience as a non-registered user, if we filled out information in a form-field, left the screen, and then started entering that same information again, previously entered information was not suggested in a dropdown, and had be typed out again which is extremely tedious. To improve this, we added a suggestions dropdown that would trigger on a user entering their information in a previously filled out form-field.
In the previous experience, almost all screens had extremely poor UI. There was a lack of hierarchy, poor use of typography, elements were overflowing out of their bounds (could be considered more of a dev issue), not enough padding between elements, poor use of colours, and not enough negative space between sections. To improve this, you can see that all the screens got a revamp to improve it (with ticketing getting the largest overhaul), and make navigating through the purchasing a ticket experience much more pleasurable.
In the previous experience, the error messages were poorly formatted which made it very difficult to parse which fields needed to be corrected, and what the correct steps were to do so. To improve this, we numbered the error messages in list format, and made sure to add how the user could rectify the issues (even though it seems pretty obvious) to reduce the cognitive load.
During the redesign phase, my focus was on overhauling Select Departure Ticket whilst my partner focused on redesigning the two screens that came after ticket selection, Trip Options and Personal Details.
Initially, I looked at other designs for ideas as to what the ticket selection screen could look like (see Inspiration) and started sketching ideas that resembled the current Air France experience with some features from the inspiration. The rationale behind this was that if the interface was similar to the previous experience, then users that used the app more regularly would be able to navigate in a familiar, but enhanced manner.
It is very surprising to me that an organization as large as Air France has a mobile app currently in the state it is. I believe this case-study demonstrates the importance of conducting a heuristics evaluation on on an existing flow/app/service/product, as it exposes many usabilities issues that result in a tremendously poor experience—an experience that visitors are currently having right now as you read this (c. July 2019).
To be honest, I feel that many of these design issues should have been identified and fixed pre-development as it will be much more be costly to fix now, but hey, we wouldn't have been able to do this case-study if it wasn't for these issues.
Reflecting on this project, I feel there were a few hurdles that we experienced while conducting the heuristic evaluation and redesign, specifically:
Initially, our reasoning was to use our own custom scoring system instead of the Severity Ratings for Usability Testing because we thought that more flexibility would make the process easier. However, due to what I consider the rather subjective nature of a heuristics evaluation (or perhaps it's just because I'm a novice evaluator), the added flexibility actually made it harder to provide a usability score for each heuristic as the lines were between scores were blurred. Additionally, our scoring system did not have the fidelity with which to prioritize the fixing of usability issues (present in the Nielsen severity ratings), which would have been helpful going into the redesign phase due to time-constraints.
In the future, if I was to perform a heuristic evaluation again, I would likely use the Nielsen severity ratings as the foundation complemented with further explanation and examples of each of the terms used, such as 'Usability catastrophe' and 'Major/Minor usability issue'. This would, therefore, give more stringent guidelines for multiple evaluators to reach a consensus and leave less room for subjectivity.
I felt that going through this evaluation process seemed quite open-ended when assessing each heuristic. As novice evaluators, we could kind of make sense of what each heuristic definition was trying to get at, but I feel a more specific set of guiding questions would have been really beneficial for us. For example, as I was writing this case-study, I started researching how other heuristic evaluations were conducted. I found this article, Heuristic analysis of yuppiechef where the author uses 2-3 guiding questions in order to evaluate each heuristic.
For example, the definition for Error Prevention is, "Prevent problems from occurring in the first place or check for them and present users with a confirmation option before they commit" and the guiding set of questions used in her analysis were,
In the future, if I was to perform a heuristic evaluation again, I'd make sure to use a guided set of questions like this in order to provide specificity and direction of the heuristic evaluation, which would ensure that all evaluators are on the same page and leave less room for subjectivity.
If you made it this far, thank you for taking the time to go through (or skim) this project. If anyone has any feedback for me regarding anything on this project, please feel free to get in touch with me.
© 2019 Ivan Rickard Liow
© 2019 Ivan Rickard Liow
© 2019 Ivan Rickard Liow