Changes when listing with our API ⚙

Developers
1

Hey folks :wave:

I’m James, and I work in the App Evolution squad at Monzo. I wanted to give you a heads up about some changes we’ve just made to our /transactions endpoint.

Why the Change?

We’ve recently been looking at improving the resilience of some of our APIs. While we typically prefer to provide notice for changes, we’ve decided to make these adjustments immediately.

What’s Changing?

We’ve made three changes to the /transactions endpoint:

  • Requests now have a default limit of 30. You can still override this up to the maximum of 100.
  • The since parameter will now default to 1 month before the before parameter.
  • We will now return a 400 status code if you request more than ~1 year of transactions in a single call.

This means that by default the /transactions endpoint will provide the smallest of the latest 30 transactions or the last months worth. Therefore, you must provide pagination parameters if you want to fetch prior transactions.

Future Adjustments

We understand that finding the right balance is crucial. As we move forward, we may fine-tune the exact values to further improve the consistency of requests to the API service. Additionally, similar restrictions may be implemented on other list endpoints if we observe similar platform issues.

If you have any questions or concerns, feel free to drop a comment!

18 Likes
3

Hello, thank you for the update.

It is surprising that this was implemented without notice. I have just stumbled across this when I spotted that latest transactions were not being returned.

I usually query my data by month, using the start and end of the month I am querying. Now, however, if there is ever more than 100 transactions a month, I will have to implement some coding changes.

A limit of 100 seems low for an outright limit. If a limit like this is essential, it would be much better if the API returned paginated URLs in the response, like most APIs do in these cases. For example, if the number of results between the specified “since” and “before” values exceeds the limit, be it the default of 30, or a custom defined value, then returning a URL to the next page would be very useful.

4 Likes
4

Thanks for the update! I was a bit confused as to why my API was no longer returning the latest transactions. Good to see you’ve also kept the docs up-to-date, so thanks for that!

Given that we now have to make (potentially) multiple API calls, when before we only had to make one, some transparency on how the rate limiting is applied would be much appreciated. I’m currently artificially limiting my requests to a maximum of 1 every 5 seconds, which may seem pretty tame, but I’ve been 429’d for a lot less than that before. What client-side rate-limiting strategy do you propose to avoid causing unexpected errors? Would it be possible to return rate-limiting information in the response headers, so that clients can act accordingly?

Also, now that pagination is mandatory, it would also be nice if the /transactions endpoint returned some pagination information, such as whether there’s a next/previous page, how many transactions/pages there are total, etc. With the new limit, if the total number of transactions is a multiple of the limit, the only way to know that you’ve reached the last page is to make an extra request, that would yield zero transactions.

2 Likes
5

I am glad I am not the only one that thinks pagination URLs would be useful. It would be amazing if this can be implemented :pray:

6

Hey both :wave:

It is surprising that this was implemented without notice. I have just stumbled across this when I spotted that latest transactions were not being returned.

Yeah we would have preferred to have given notice for this change but we prioritised reducing the impact we were seeing when not limiting these requests. Sorry for the disruption this caused as we understand it’s not ideal for you.

A limit of 100 seems low for an outright limit.

Most of our customers will have far less than 100 transactions per month but I understand that you’d want to handle pagination anyway so your code doesn’t break if you happened to have more than that in a particular month.

Given that we now have to make (potentially) multiple API calls, when before we only had to make one, some transparency on how the rate limiting is applied would be much appreciated. I’m currently artificially limiting my requests to a maximum of 1 every 5 seconds, which may seem pretty tame, but I’ve been 429’d for a lot less than that before.

The rate limiting is applied as a maximum per period of time. If you’ve been 429’d despite going at a slow rate before then it could be we allowed X per 24 hours to that particular endpoint for example which means your “burst” rate is less important. As far as I know you should be able to make at least one request per second to this endpoint. Let me know if this isn’t the case :pray:

What client-side rate-limiting strategy do you propose to avoid causing unexpected errors? Would it be possible to return rate-limiting information in the response headers, so that clients can act accordingly?
Also, now that pagination is mandatory, it would also be nice if the /transactions endpoint returned some pagination information, such as whether there’s a next/previous page, how many transactions/pages there are total, etc. With the new limit, if the total number of transactions is a multiple of the limit , the only way to know that you’ve reached the last page is to make an extra request, that would yield zero transactions.

I think it should be good to just handle a 429 with an exponential backoff.

Thanks for the feature suggestions, it would be really useful to understand how many developers feel the same way. We can add tickets to investigate these in our backlog but we would only want to invest in changes where we know they will have a positive impact to a wide portion of our community. :pray:

4 Likes
7

The latest transactions I am only able to fetch is 23rd and prior… anything later is now not being returned. Where am I going wrong here?

     var getTransactions = (await client.GetTransactionsAsync(accounts[0].Id, expand: "merchant",
                        new PaginationOptions
                        {
                            Limit = 100,
                            // SinceTime =  date.AddMonths(-1),
                            SinceId = null,
                            // BeforeTime = date
                        }))
                    .OrderByDescending(x => x.Created)
                    .ToList();
8

Hello, Karim.

Is there a reason your dates strings are commented out? If you look at what is stated as part of this new thread’s initial post:

  • The since parameter will now default to 1 month before the before parameter.

To me, this seems to indicate that you at least need to include the before parameter.

Try setting before to tomorrow.

9

Hi, how can we find the first transaction if only sinceId is supported? Then walk forward and retrieve all transactions after.

10

Three suggestions.

  1. Add an option to change the direction of the data asc/desc
  2. Add beforeId option
  3. Add next / prev page links in returned metadata
11

These changes are not fit for purpose :man_facepalming:.

  1. A raw request to retrieve transactions without any options returns 30 transactions from a couple of months ago. You would expect this to be the last 30 transactions.

  2. For accounts with lots of transactions, setting “before” to “today” does not return the latest transactions, the only way to retrieve today’s transactions is if you walk forward using “since”. This is an ugly extra step if you’re trying to retrieve all transactions for an account.

  3. There is no asc/desc option or beforeId option so you have to use the “before” time which introduces a v slim chance that you may miss transactions or get stuck in a loop if for some reason you have more transactions than the transaction limit for 1 timestamp.

  4. Because there is a cap on the time range, programmatically walking back through transactions isn’t guaranteed to retrieve all transactions if an interval between transactions exceeds this gap.

12

This is completely broken :man_facepalming:. Looping back through transactions by providing a before id id does not return results of first page (earliest transactions)

15

Brute force method:

  • starts on monzo epoch
  • uses sinceId when available
  • falls back to walking through each month
const getFullTransactionHistory = async () => {
  const getTransactions = async (since: string, before: string): Promise<any[]> => {
    const query = new URLSearchParams({ limit: "100", account_id: this.accountId, since, before });
    const options = { headers: { Authorization: `Bearer ${this.accessToken}` } };
    const response = await axios.get<{ transactions: any; has_more: boolean }>(
      "https://api.monzo.com/transactions?" + query,
      options
    );
    return response.data.transactions;
  };

  let sinceTime = new Date("2015-02-18");
  let beforeTime = addMonths(new Date(sinceTime), 1);
  let sinceId: null | string = null;

  let transactions: any[] = [];
  let complete = false;

  console.log("Getting monzo transactions");

  while (!complete) {
    try {
      const pageTransactions = await getTransactions(
        sinceId || sinceTime.toISOString(),
        beforeTime.toISOString()
      );

      transactions = [...transactions, ...pageTransactions];

      if (pageTransactions.length > 0) {
        sinceTime = pageTransactions[pageTransactions.length - 1].created;
        beforeTime = addMonths(new Date(sinceTime), 1);
        sinceId = pageTransactions[pageTransactions.length - 1].id;
      } else if (startOfDay(new Date(sinceTime)) < startOfDay(new Date())) {
        sinceTime = addMonths(new Date(sinceTime), 1);
        beforeTime = addMonths(new Date(sinceTime), 1);
        sinceId = null;
      } else {
        complete = true;
      }

      console.log(
        `Scanning ${format(new Date(sinceTime), "yyyy MM")}, found ${
          transactions.length
        } transactions, First transaction: ${transactions[0]?.created}, Last transaction: ${
          transactions[transactions.length - 1]?.created
        }`
      );

      await new Promise((resolve) => setTimeout(resolve, 100));
    } catch (error) {
      if ((error as any)?.response?.data?.code === "bad_request.invalid_time_range") {
        console.log(`caught: bad_request.invalid_time_range`);
        sinceTime = addMonths(new Date(sinceTime), 1);
        beforeTime = addMonths(new Date(sinceTime), 1);
        sinceId = null;
      } else {
        throw error;
      }
    }
  }

  console.log(`Successfully retrieved transactions. Found ${transactions.length}.`);

  return transactions;
};
16

Just had to register to respond to the changes in transaction listings.

Wow guys this is really badly thought through. Do you guys do dogfooding? I.e do you try to use your own API for simple tasks like
a) last updates since X
b) all of the transactions?
The amount of weirdness to overcome to do those two simple actions just makes me go “meh” and consider giving up altogether.

17

Hey, @davi ,

Thanks for sharing. Other than the little annoyances I reported, it is really simple to do both of these tasks.

Do you have some code to share? I am sure we can point you in the right direction of where you are going wrong :blush:.

For b), though, you will need to set up some sort of iteration until you have all of the data, due to the recent changes. To get data since X, you can simply use heir sinceTime and beforeTime parameters.

Anyway, as I say, share your code and one of the community will help to get you on your way.

18

Hey @kingridgehead - thanks for your reply.

My grievances:

  • when requesting a block of transactions which exceeds the limit, there is no error. Payload is truncated silently;
  • when paginating, there is no way of knowing which beforeTime to use to get the next 100 transactions;
  • if using too small window, you hit rate limits. If you use too large window - see above, you get subset of requested data;

It’s not that I’m incapable of doing it right - I integrated a lot of APIs in my past life as a software dev. It’s just I can’t figure out what’s “right” in this design. Sigh.

19

Hello again, @davi. I get you.

Their API certainly isn’t ideal is it, lol? I guess it depend on how you are accessing your data. One of the things I mentioned in my original post here was the lack of pagination.

If it helps, in the UI I have set up, I access my data on a monthly basis, which works perfectly as, in most cases, I do not tend to have more than 100 transactions per month.

So, to access a month’s worth of data, I use the URL:

> https://api.monzo.com/transactions?
> account_id=ACCOUNT_HERE&
> since=2024-03-01T00:00:00+00:00&
> before=2024-04-01T00:00:00+01:00&
> limit=100&
> expand[]=merchant

If you generally have more than 100 transactions in a month, you will obviously need to do more than this, and probably do something annoying like select select smaller subsets, due to their lack of paginated URLs.