Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MongoError: Index with name: username_1 already exists with different options #11666

Open
kentonv opened this issue Oct 3, 2021 · 12 comments
Open

Comments

@kentonv
Copy link
Contributor

@kentonv kentonv commented Oct 3, 2021

Starting with Meteor 2.4, the _ensureIndex() method has different behavior: Previously, if the index already existed, regardless of options, then the call did nothing. Now, if the options differ, then the call throws an exception.

For new apps, the new behavior is clearly preferable.

Unfortunately, with Sandstorm, we've discovered that many of Meteor's own index-creation calls -- including the ones in accounts-base -- fail when run against older databases. It appears that Meteor used to create indexes with an option "safe": true, but no longer does. Mongo sees this as a change in options, and so all of Meteor's own calls to createIndex() fail if the indexes were originally created using a sufficiently old version of Meteor. This means the app cannot start unless the user manually goes and deletes all these indexes.

Note that the "safe" option was removed from Mongo quite some time ago, which may mean that users don't see this issue unless they are using old versions of Mongo. Sandstorm is, sadly, stuck on Mongo 2.6. The reason for this is, updating to a newer version of Mongo apparently requires some manual intervention to migrate data. Sandstorm is installed on thousands of individual users' machines where it has been auto-updating for over 7 years. Many of those users are non-technical, and we cannot ask them to take manual action to update their database. In order to update, we would need to build a fully-automated Mongo migration that happens in the background of live servers, which seems like a large engineering effort that we don't have time for.

I am not sure what to do about this. I think ideally, instead of failing when options differ, we'd drop and recreate the index with the correct options. But we need Meteor itself to do this with its own indexes. We have no way of knowing the full list of indexes that may need to be rebuilt, so it's hard for me to work around this.

@kentonv
Copy link
Contributor Author

@kentonv kentonv commented Oct 3, 2021

In one particularly-old Sandstorm instance I looked at, I found the following indexes on Meteor.users had "safe": true:

username_1, emails.address_1, services.resume.loginTokens.hashedToken_1, services.resume.loginTokens.token_1, services.resume.haveLoginTokensToDelete_1, services.resume.loginTokens.when_1, services.github.id_1, services.google.id_1

@StorytellerCZ
Copy link
Contributor

@StorytellerCZ StorytellerCZ commented Oct 3, 2021

Wow, that is a really old version of MongoDB. I don't even see the safe option in MongoDB 2.6 docs. Looking through recent commits I don't see any change that would indicate that we had changed anything about setting index options, so it might be some old MongoDB driver default. 🤔

@kentonv
Copy link
Contributor Author

@kentonv kentonv commented Oct 3, 2021

Yes, it seems like the Mongo driver used to set "safe" by default, or something. AFAICT this option can't even be set explicitly with the current API / driver -- it's ignored when I try to set it, which unfortunately makes it kind of hard to reproduce the problem outside of production. Note that I'm not even 100% sure that the "safe" option is the problem, but it's the only apparent difference I can see between old indexes and new ones.

I think what would work for me is if there were some flag or environment variable or something that we could set that would cause createIndex and _ensureIndex to automatically delete and recreate indexes when the options differ, and log a warning to stderr.

@ocdtrekkie
Copy link
Contributor

@ocdtrekkie ocdtrekkie commented Oct 3, 2021

It looks like the "safe" option was removed in #4340 (in 2015) after being added in 2012. So I'd gather any Meteor app servers spun up in those three years or so, and still running, should have this problem?

@kentonv
Copy link
Contributor Author

@kentonv kentonv commented Oct 3, 2021

@ocdtrekkie Good find.

What's unclear to me is, if someone is not stuck on Mongo 2.6 like we are, but rather is using a newer version of Mongo, will they still hit this problem? Or do newer versions of Mongo ignore / drop this flag (which apparently never meant anything in the first place)?

@menewman
Copy link
Contributor

@menewman menewman commented Oct 4, 2021

It looks like the "safe" option was removed in #4340 (in 2015) after being added in 2012. So I'd gather any Meteor app servers spun up in those three years or so, and still running, should have this problem?

What's unclear to me is, if someone is not stuck on Mongo 2.6 like we are, but rather is using a newer version of Mongo, will they still hit this problem? Or do newer versions of Mongo ignore / drop this flag (which apparently never meant anything in the first place)?

One more data point: We have a Meteor application that's been running since (I think) 2014, and we have indeed seen this type of index error on upgrade to Meteor 2.4. Our current MongoDB version is fairly recent at 4.4+.

It was a pretty bad/subtle bug, because it didn't happen with local and testing environments (which were created more recently) but only on our production database. Thankfully, errors on startup are handled fairly gracefully within Galaxy and we were able to rollback to the old version easily while resolving the index errors.

For conflicts like this in our own code, we switched from using _ensureIndex (or createIndex) to a monkey-patched ensureIndex method that catches this error and removes/recreates the index with updated options if needed.

(This monkeypatch is available in this package: https://github.com/CodeSignal/meteor-protomongo)

Unfortunately, that doesn't really help if the index is being added in core Meteor or Accounts package code, where we don't have control over the index creation. To fix this, we had to resort to the manual fix described in the original issue above: removing the index from the DB before activating the new version so that it can be created again with the right options.

Perhaps a similar tweak could be used within Meteor/Accounts code to make sure conflicting options are overwritten when creating these core indexes?

@Gywem
Copy link

@Gywem Gywem commented Oct 26, 2021

It looks like the "safe" option was removed in #4340 (in 2015) after being added in 2012. So I'd gather any Meteor app servers spun up in those three years or so, and still running, should have this problem?

I can confirm this. The project I worked on started around these years and upgrading to 2.4 has introduced an issue with conflicted indexes and the options as reported here.

removing the index from the DB before activating the new version so that it can be created again with the right options.

This is what would work for us as well.

@Gywem
Copy link

@Gywem Gywem commented Oct 26, 2021

At the end the problem in the project I mentioned was not exacly the same. But with the usage of new createIndex function, one of our indexes was expected to be the same as the Meteor core packages expects, but it isn't. Like emails.address in our case is not unique, but meteor accounts core expects to be unique.

To avoid that we have altered the createIndex behavior to skip the "creation" of that index in particular, to avoid the error thrown around the mismatch.

I believe the same can be done in the other scenarios and avoid that index to be touched. But eliminating it and leave the update to restore it again is valid as well.

If anyone has the same problem as we did, you have here an snippet to avoid any particular creation of an index that meteor core uses.

const SKIP_INDEX_CREATION = ['emails.address'];

const originalCreateIndex = Mongo.Collection.prototype.createIndex;
Mongo.Collection.prototype.createIndex = function createIndex(...args) {
  const indexName = args?.[0];
  if (
    indexName &&
    typeof indexName === 'string' &&
    SKIP_INDEX_CREATION.includes(indexName)
  ) {
    return null;
  }
  return originalCreateIndex.apply(this, args);
};

If you still have the problem make sure you have this instantiated before any other meteor core package by adding this code within a package that is loaded first on .meteor/packages file.

@andreasheim
Copy link

@andreasheim andreasheim commented Dec 31, 2021

Ran into this issue as well. Just want to point out it may be more important with which version (of Mongo or Meteor) the app was created, not what it is currently on. Our app is on Mongo 4, and I was unable to deploy with Meteor 2.4 or 2.5.

Interestingly, it's not an issue locally, with a recent prod backup, nor on our staging env, which was created about a year ago with a prod Mongo backup.

@kentonv
Copy link
Contributor Author

@kentonv kentonv commented Jan 12, 2022

@StorytellerCZ it appears this affects everyone who has Meteor apps that have been running continuously for many years, even if their database is up-to-date.

Can Meteor add some code to automatically detect this problem and work around it, by either fixing the flags on the index or rebuilding it? It's very hard for us to do this at the application level since the failure happens at Meteor startup before the app gets a chance to run any code.

@StorytellerCZ
Copy link
Contributor

@StorytellerCZ StorytellerCZ commented Jan 12, 2022

@kentonv I'll try to figure out something. Could you give me a sample how the index looks like in MongodDB when inspected so that I could replicate it locally for testing?
I'm thinking that we would detect this specific index, write out a warning that it should be fixed manually, but then make sure that the app continues building and working. Thoughts?

@kentonv
Copy link
Contributor Author

@kentonv kentonv commented Jan 13, 2022

@StorytellerCZ as an example, I have this index on db.users:

	{
		"v" : 1,
		"key" : {
			"services.resume.loginTokens.token" : 1
		},
		"unique" : true,
		"ns" : "meteor.users",
		"name" : "services.resume.loginTokens.token_1",
		"safe" : true,
		"sparse" : 1
	}

AFAICT the "safe": true part causes the problem. All my very-old indexes have this flag.

kentonv added a commit to sandstorm-io/sandstorm that referenced this issue Jan 17, 2022
See: meteor/meteor#11839

This was fixed in a newer Meteor. However, we currently can't upgrade Meteor due to a different problem: meteor/meteor#11666
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants