Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

concurrent inserts cause an index to be deleted #1619

Closed
abdollar opened this issue Jan 18, 2012 · 10 comments
Closed

concurrent inserts cause an index to be deleted #1619

abdollar opened this issue Jan 18, 2012 · 10 comments

Comments

@abdollar
Copy link

I have an application that performs concurrent inserts on an elasticsearch index and is using 4 cpus on a cluster with 5 nodes using 0.18.5 - a single node has 16 cores and inserts are being performed on the node with the elasticsearch instance.

At a certain point in time (and its not that predictable) I see remove_mapping in the logs and a new index is being referenced when using count.

If I force/bind the application that performs the inserts to one cpu the problem goes away ; as in the index does not get rebuilt and index counts don't get reset and I don't see remove_mapping in the logs
@kimchy
Copy link
Member

kimchy commented Jan 18, 2012

Remove mapping is only being called when you actually do curl -XDELETE localhost:9200/index/type, or you execute a delete mapping Java API. There is no other way to do it. Do you delete data in your indexing code? Also, what do you mean a new index being referenced? Can you share your indexing code?

@abdollar
Copy link
Author

I do not delete data in my indexing code. Its just performing inserts. The application that performs the inserts is an http server written in go.

If I do call delete explicitly I can see it happen in the logs : like so

[2012-01-18 01:40:35,920][INFO ][cluster.metadata ] [Tag] [users] deleting index

However, I am not seeing this. I am seeing this

[2012-01-18 00:11:32,057][INFO ][cluster.metadata ] [Tag] [[users]] remove_mapping [user]
[2012-01-18 00:11:32,110][INFO ][cluster.metadata ] [Tag] [users] update_mapping user
[2012-01-18 00:11:32,500][INFO ][cluster.metadata ] [Tag] [users] update_mapping user
[2012-01-18 00:11:32,779][INFO ][cluster.metadata ] [Tag] [users] update_mapping user
[2012-01-18 00:23:23,058][INFO ][cluster.metadata ] [Tag] [users] update_mapping [user]
[2012-01-18 00:24:59,086][INFO ][cluster.metadata ] [Tag] [[users]] remove_mapping [user]
[2012-01-18 00:24:59,106][INFO ][cluster.metadata ] [Tag] [users] update_mapping user
[2012-01-18 00:24:59,442][INFO ][cluster.metadata ] [Tag] [users] update_mapping user
[2012-01-18 00:24:59,734][INFO ][cluster.metadata ] [Tag] [users] update_mapping user
[2012-01-18 00:40:11,283][INFO ][cluster.metadata ] [Tag] [[users]] remove_mapping [user]
[2012-01-18 00:40:11,306][INFO ][cluster.metadata ] [Tag] [users] update_mapping user
[2012-01-18 00:40:11,683][INFO ][cluster.metadata ] [Tag] [users] update_mapping user
[2012-01-18 00:40:12,460][INFO ][cluster.metadata ] [Tag] [users] update_mapping user
[2012-01-18 00:54:25,358][INFO ][cluster.metadata ] [Tag] [users] update_mapping [user]
[2012-01-18 00:56:26,375][INFO ][cluster.metadata ] [Tag] [[users]] remove_mapping [user]
[2012-01-18 00:56:26,389][INFO ][cluster.metadata ] [Tag] [users] update_mapping user
[2012-01-18 00:56:26,770][INFO ][cluster.metadata ] [Tag] [users] update_mapping user
[2012-01-18 00:56:27,054][INFO ][cluster.metadata ] [Tag] [users] update_mapping user
[2012-01-18 01:01:26,416][INFO ][cluster.metadata ] [Tag] [users] update_mapping [user]
[2012-01-18 01:06:29,784][INFO ][cluster.metadata ] [Tag] [[users]] remove_mapping [user]
[2012-01-18 01:06:29,808][INFO ][cluster.metadata ] [Tag] [users] update_mapping user
[2012-01-18 01:06:30,150][INFO ][cluster.metadata ] [Tag] [users] update_mapping user
[2012-01-18 01:06:30,420][INFO ][cluster.metadata ] [Tag] [users] update_mapping user

Each time remove mapping is called it seems the old index is deleted and a new index is created.

If I change the code in the http server from using 4 cpus and change it to use 1 cpu - the problem goes away - as in I dont see remove_mapping getting called and index keeps getting bigger.

In go I do this to force it to use 1 cpu.

runtime.GOMAXPROCS(4)
to
runtime.GOMAXPROCS(1)

@kimchy
Copy link
Member

kimchy commented Jan 18, 2012

I am not talking about deleting an index, but deleting a mapping. In your case, for an index called users, and a mapping type called user, someone called curl -XDELETE localhost:9200/users/user. Thats the only code base that can cause this logging message to appear. Its not called as part of the indexing process...

@abdollar
Copy link
Author

Hmm - Thats strange. From my understanding there are no DELETE requests going to elasticsearch - just PUT requests

I will see if I can reproduce this with a simpler example.

@kimchy
Copy link
Member

kimchy commented Jan 18, 2012

Here is a custom build: http://dl.dropbox.com/u/2136051/elasticsearch-0.18.8-SNAPSHOT.zip that logs when delete mapping is called, can you run it with it and see if you see the log. The logging is INFO level and states: "<------ DELETE MAPPING CALLED!".

@abdollar
Copy link
Author

OK. I updated all the nodes on the cluster to the custom build - elasticsearch-0.18.8-SNAPSHOT.zip
Reset the concurrency to 4.
So far I don't see any log messages at all related to mappings.
It'll take around 3 days to ingest everything at the current rate.
I'll keep an eye on the index size and logs and will keep you posted.

@kimchy
Copy link
Member

kimchy commented Jan 19, 2012

I assume that you don't see the remove_mapping [user] message as well?

@abdollar
Copy link
Author

Yep. I don't see any log messages related to mappings. I don't see the remove_mapping message. It's been running happily for quite some time now - as in the index is growing and if I search for id:1 I can still see it.

@kimchy
Copy link
Member

kimchy commented Jan 19, 2012

Maybe its the 0.18.5 version? Thats weird, since there wasn't a problem like that. Well, if it works, you can run with 0.18.7 since the 0.18.8 snap you have has almost no changes on top of it.

@abdollar
Copy link
Author

Thanks. Currently at 15 million records and climbing. I don't see any issues. Closing

martijnvg pushed a commit to martijnvg/elasticsearch that referenced this issue Jan 31, 2018
@pickypg @martijnvg
Always Accumulate Transport Exceptions (elastic#1619)
1cef531 
This is the x-pack side of the removal of `accumulateExceptions()` for both `TransportNodesAction` and `TransportTasksAction`.

There are occasional, random failures that occur during API calls that are silently ignored from the caller's perspective, which also leads to weird API responses that have no response and also no errors, which is obviously untrue.
martijnvg pushed a commit that referenced this issue Feb 5, 2018
@pickypg @martijnvg
Always Accumulate Transport Exceptions (#1619)
4f866d9 
This is the x-pack side of the removal of `accumulateExceptions()` for both `TransportNodesAction` and `TransportTasksAction`.

There are occasional, random failures that occur during API calls that are silently ignored from the caller's perspective, which also leads to weird API responses that have no response and also no errors, which is obviously untrue.
martijnvg added a commit that referenced this issue Feb 5, 2018
@martijnvg
Merge remote-tracking branch 'es/6.x' into ccr-6.x
ff62d14 
* es/6.x: (155 commits)
  Make persistent tasks work. Made persistent tasks executors pluggable.
  Removed ClientHelper dependency from PersistentTasksService.
  Added AllocatedPersistentTask#waitForPersistentTaskStatus(...) that delegates to PersistentTasksService#waitForPersistentTaskStatus(...)
  Add adding ability to associate an ID with tasks.
  Remove InternalClient and InternalSecurityClient (#3054)
  Make the persistent task status available to PersistentTasksExecutor.nodeOperation(...) method
  Refactor/to x content fragments2 (#2329)
  Make AllocatedPersistentTask members volatile (#2297)
  Moves more classes over to ToXContentObject/Fragment (#2283)
  Adapt to upstream changes made to AbstractStreamableXContentTestCase (#2117)
  Move tribe to a module (#2088)
  Persistent Tasks: remove unused isCurrentStatus method (#2076)
  Call initialising constructor of BaseTasksRequest (#1771)
  Always Accumulate Transport Exceptions (#1619)
  Pass down the provided timeout.
  Fix static / version based BWC tests (#1456)
  Don't call ClusterService.state() in a ClusterStateUpdateTask
  Separate publishing from applying cluster states
  Persistent tasks: require allocation id on task completion (#1107)
  Fixes compile errors in Eclipse due to generics
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kimchy @abdollar