Verification overhaul

Revamping the verification process to improve its effectiveness and efficiency.

Product Uber Carshare

Role Product Manager

Timeframe February 2024 - September 2024

Results Reduced fraud cases and bad actors being onboarded on to the platform by 18% over a 7 month period while improving application processing time by 7.5% through automation, introducing new checks, removing redundant checks, and recalibrating existing processes

Background

Uber Carshare’s member verification process had been built up over ten years as the business grew, incorporating new checks as they became necessary through arising problems.

This resulted in a verification process with a long processing time, requiring agents to have extensive knowledge in individual checks. Training a new agent through this process had also become a challenge and took two months on average.

We had started to see high rates of theft, damage, late returns, arrears and speeding when we rebranded from Car Next Door to Uber Carshare in August 2022. This continued and worsened as we launched into United States and Canada in September 2023.

In addition to this, Uber Carshare is a peer to peer platform with car owners and car borrowers, but Carshare only had the one verification flow and all members were asked to complete this flow. We ended up with issues like car owners going through and completing driving eligibility when it was not required.

My role

As a Product Manager in the Risk squad, my role was to tackle the following problems for all regions:

  • Bad actors on the platform are stealing, damaging, speeding and returning cars late. This often led to arrears that eventually turned to bad debt. This had been steadily increasing over the years, especially after the rebrand to Uber Carshare.

  • Verification process is complex and requires a substantial time to process, it also requires a lot of training hours. This also led to new borrowers wanting to borrow cars quickly but were not approved on time for their trip. Majority of borrowers booked cars 1-2 hours prior to their booking start time, our verification SLA was 3 hours though most applications were approved well before.

  • All members are required to go through the verification flow even if some of the checks do not apply to them.

I started by breaking down the verification process into individual steps, removing legacy processes and adding in new checks, rearranged the way in which they were conducted. I then started improving on individual checks within the verification process. I created an ideal future state and worked back to MVP and prioritised according to business metrics. I conducted business analysis, research, user interviews and shadowed agents processing applications, managed stakeholders, met with various vendors, and worked closely with internal agents and the engineering team.

The process

1. Understanding the verification process

First step in this process was to break down what we did and why we did them. Speaking to various parts of the business, I created a sheet which outlined all the checks and which region

This process revealed that all checks belongs to three different categories.

  1. Identity fraud checks

  2. Credit eligibility checks

  3. Driving eligibility checks

Once we bundled the checks into each group, we found that some checks were perhaps not necessary or they provided little value.

This process also highlighted checks we were missing, especially for US and Canada. This was a result of the speed at which Uber wanted to expand into new regions with the ‘launch now, fix later’ approach.

2. Ideal future state

To get a sense of where we want to be with verification process in the future, we started to breakdown what verification meant for Carshare and how it might be used in the future.

Below is the future state we felt would improve the verification processing speed and improve cost per member since it removes the need to run every check in every member.

In the ideal state of verification, checks are divided into sections. If a new member wants to borrow a car, they go through the necessary checks in the respective country, for example:

  1. Identity fraud checks

  2. Credit eligibility checks

  3. Driving eligibility checks

If a member only wants to list their car on the platform, they only need the “Identity fraud checks” because they won’t be driving any cars on the platform and will only be earning money, removing vendor costs and speeding up the process.

In the example highlighted in blue, an existing owner member wants to borrow cars in Australia, this member will now need to complete the “Australia - Credit eligibility checks” and “Australia - Driving eligibility checks” to be eligible to drive in Australia.

In the example highlights in orange, an existing borrower approved in Australia wants to borrow cars in the United States, much like the above example, they will now need to complete the “United States - Driving eligibility checks” to be eligible to drive in the United States. We do not need to conduct the credit eligibility in the US for this member since we are already confident in their ability to pay for the service.

Before starting on developing the ideal state, we wanted to clean up the current processes and improve the checks to minimise bad actors on the platform, these were the steps taken:

  1. Remove any checks that are unnecessary or is doing the same job.

  2. Automating checks that can be automated.

  3. Adding in new checks to fill the gaps identified.

  4. Improving checks that are not working quite right.

  5. Simplify the agent process by cleaning up the admin page and creating new guides that are easy to follow.

Once the above was completed, we planned on initiating the work to split the current verification flow into various flows to match the ideal state.

3. Removing legacy checks

With the years of adding in various checks along the way, first task was to remove the unnecessary checks from the code and admin pages. These elements bloated the admin page adding cognitive load and created confusion during new agent training.

Here are some examples of elements we removed from the page:

  1. Outdated banners and pop ups that were no longer valid.

2. Legacy checks that no longer applied but still showed up on the page.

3. Complicated results from Experian Credit report - agents were expected to go over each report for any red flags but ultimately we relied on the overall score to make a decision.

4. Filling the gaps and tightening the checks

The initial step of mapping out the current checks also highlighted some gaps in the verification we had.

  • Document verification service (DVS) in Australia had stopped due to Experian’s offering changing in March 2023. This meant that we had no idea if the licences that were being submitted by users were a legitimate licence recognised by the Australian Government.

  • In US and Canada, we did not have a robust driving eligibility check. We knew we could access driving history including violations, convictions, suspensions, accidents and any outstanding fines in United States and Canada, very valuable information for Carshare to determine user’s ability to return the borrowed cars safely.

  • We were aware of Uber’s own system that flagged bad actors on the Uber ride and eats platforms. It was crucial for Carshare to have this information to determine a users eligibility and also send bad actor information back to Uber for better overall ecosystem.

Document verification service (DVS) in Australia

We reached out to multiple vendors who were able to provide Carshare with the DVS checks. We ended up with a vendor who was able to provide us immediate access to a manual portal which allowed Carshare agents to login, input the driver’s licence details manually, then get the DVS results. This allowed us to start immediately and the integration to automate this process was put in the backlog. We still felt automating process was quite important since it would cut down on processing time and remove human error from the check.

Motor Vehicle Registry (MVR) in United States and Canada

We reached out to multiple vendors in North America who could provide us with a service called MVR. Much like DVS provider in Australia, we were able to sign on with a vendor who had a manual portal with a view to automate it in the future. Since this was a new service for our agents, we created flows and confluence pages that outlined the process.

Uber flagged user check

This was a very valuable check for Carshare. Uber had a database of bad actors, tagged into various categories from banned members, fraudulent credit card users, late payments and much more. This database was large and robust, allowing us to fine tune the level of risk we were willing to take with new users. Again, we approach this in two step, MVP version was to upload a full list of bad actors onto our platform, create a button in the admin page that conducted this check. Second phase was to integrate with the database and have the system automate the check at the time of application submission by the user.

Below shows blocked users with this check over a 7 week period, each of these bad actors have the ability to cause thousands of dollars worth of damage. If they ride off a car, it could be as high as AUD$60,000 excluding operational costs.

5. Improving application processing for agents

Amongst various processes that agents were tasks to do everyday, application processing was by far the most time consuming, with a dedicated team working just on applications 24/7. In an ideal world, the whole application reviewing process would be automated but in order to get closer to that stage, we needed to think about how we wanted to use the checks as a business.

Verify sooner flow

One of the key points we discovered while interviewing agents and shadowing them reviewing applications, was that they were required to go through every single check even if they knew the application would be rejected. This meant a huge amount of time was wasted on bad applications. To solve this problem, we need to identify which checks are the most important and allow agents to reject applications without the need to do every check. This is the flow we came up with.

By breaking down the checks into four sections and allowing agents to reject after each section, we improved the application processing time significantly. To improve the user experience for the agents, we also reordered the application processing admin page to align with the sections shown above. This allowed agents to gradually work down the page instead of frantically scrolling up and down while they tried to find the information on the page.

Since this was a major update to the processes that had been in place for a long time, we created confluences pages, training videos and setup meeting and help sessions to guide the agents along the way. We also tested with senior agents first and went through a few minor iterations prior to development.

Here are some examples of documents created to help agents with the new process.

Verify sooner agent flow chart

This was a step by step guide for the agents to follow and to create alignment within the team.

Here is a close up sample of the flow above

Manager to review / Team leader hand over flow chart

We also found some ambiguity around how and when to get help from managers or team leaders, we created this flow to help agents define this process a little more.

6. Improving individual checks

In parallel to improving the verification processes, we worked on improving the individual checks with the aim to remove complexities from the overall process.

Credit score automation

Credit score was something we relied on heavily to determine a user’s ability to pay and reduce debt. Agents were asked to review the whole report in order to make a decision which was a very daunting task as the report included a lot of information.

I started by understanding the scoring, speaking to our vendor Experian, I was able to get a good understanding of the credit score in Australia. This gave us a new credit score cut off point of 600 instead of the previous 400, we then automated this so that all applications below 600 score would not even get to the manual review stage. This removed one more step from the verify sooner flow. Below illustrates the new verify sooner flow.

After monitoring this credit score automation for 2-4 weeks, we noticed that it was potentially blocking good users. We decided to manually review 150 users and started to see a pattern emerge. This showed us that instead of focusing on the credit score itself, we should be looking at Active Defaults, adverse events on file, court actions, bankruptcy act action, and if they had an active credit provider. We saw that this was a much better indicator of how a person may behave on the platform and their ability to pay for the service.

My initial action was to create a manual process for the agents to follow right away, then created and epic to automate this whole process.

IP Quality score

While reviewing some cases manually I noticed a strange occurrence where IP Quality score check was automatically blocking anyone with an invalid email address. While this makes sense logically, it also meant that we were rejecting anyone who accidentally type their email incorrectly. We were seeing around 3-5 users weekly being auto declined due to this issue.

My immediate actions was to remove this decline automation on Carshare side and started working with the vendor to find out what was the best way forward. Much like Credit score, our aim was to breakdown how the system works and how best to utilise it for our platform.

SIFT score

SIFT is a similar service to IP Quality where it runs checks on the users IP address. In June 2021, Apple released a iCloud Private Relay. Unbeknownst to us, this meant that SIFT was now flagging a lot of Apple users as suspicious or high risk. This is because iCloud private relay hides IP addresses by bundling them together making it seem like they are all using the same IP address.

We were not aware of this until we started to look deeply into SIFT and it’s usefulness in 2024. Once we realised how unreliable this check actually was for Carshare, we decided to build our own system using Amazon Rekognition and matching emails, phone, name, driver’s licence number and other data points.

Results

  • Over a 7 month period, we saw an 18% reduction of fraud cases/bad actors being onboarded.

  • Improved application processing time by 7.5% saving 25 hours per week by automating onboarding decisioning through the use of credit checks.

  • Introduced checks like Document verification service (DVS) in Australia and Motor Vehicle Registration (MVR) in US and Canada to improve the verification process.

  • Improved and recalibrated existing checks like IP Quality and SIFT checks to suit the current business objectives.

  • Became complaint with Canadian insurance vendor by conducting Motor Vehicle Registration (MVR) checks for all Canadian licence holders regardless of their location (applying outside of Canada).

Reflection

  • This was a big project and a huge area to tackle. By breaking down the problems, it allowed me to see how it could be tackled and which to prioritise first. Having an ideal future state also allowed me to keep the end goal in mind to ensure we weren’t making any changes that we’d need to revert later down the track.

  • Shadowing agents processing applicants was very interesting and informative. It allowed me to see things that were inefficient and areas for easy wins. I could also see issues that were not noticed by the agents themselves because they had become so used to the inefficient process.

  • I found that just because the current process conducts certain checks, doesn’t necessarily mean they are valid or useful checks. By asking questions around why we had them in the first place, opened discussions about what it was that we were trying to achieve.

See more of my work: