Cloudflare enabled on API endpoints #25

cerinoligutom · 2022-05-22T14:23:54Z

Try this url on an http client: https://nhentai.net/api/gallery/177013

Zekfad · 2022-05-24T14:03:26Z

That's site issue for sure.
The only thing we can do is to pass cookies into agent.
I can try to add some example solutions (e.g. try, catch cf, get token, retry with token).
That's really tricky, because CF made to prevent scrapping/boting/etc/, which is what we're doing.

Chaoray · 2022-05-29T15:42:14Z

sometimes api works but most time it doesn't. I hope api can be work again cuz this is the most comfortable nhentai api in my mind <3

Zekfad · 2022-05-30T17:15:17Z

Current workaround:
You'll need tough-cookie and http-cookie-agent for this.

// index.mjs
import { API, TagTypes, } from 'nhentai-api';
import { CookieJar } from 'tough-cookie';

import _httpCookie from 'http-cookie-agent/http';
const { HttpsCookieAgent: CookieAgent } = _httpCookie;

const jar = new CookieJar();
const agent = new CookieAgent({ cookies: { jar, }, });

jar.setCookie('cf_clearance=4iV5fuQ0.prnWCqcZ.kqLFV6hp.QQ5y_lRkEw97N3ZA-1653929202-0-150', 'https://nhentai.net/');

const api = new API({ agent, });

api.getBook(177013).then(book => {
	console.log(book.title.pretty);
});

To get cf_clearance cookie:

Get module User-Agent:
- Run node -p "(new (require('nhentai-api').API)()).getBook(-1).catch(e => console.log(e.httpResponse.req.getHeaders()['user-agent']));"
Open https://nhentai.net/api/galleries/search in browser.
Open DevTools and set User Agent to module's:
- Open bottom panel if it's not present (ESC).
- Under three dots (⋮) open Network conditions.
- Set custom User Agent to the module's:
  
  Screenshot
Without closing DevTools reload page by pressing to URL and pressing enter (needed to refresh headers)
Wait for Cloudflare to complete.
Read cookie from Network tab and request (make sure you're watching at request with your User-Agent, on screenshot you can see it as the last header):

Screenshot
Place cookie to code.

Chaoray · 2022-05-31T12:02:59Z

Current workaround: You'll need tough-cookie and http-cookie-agent for this.
// index.mjs
import { API, TagTypes, } from 'nhentai-api';
import { CookieJar } from 'tough-cookie';

import _httpCookie from 'http-cookie-agent/http';
const { HttpsCookieAgent: CookieAgent } = _httpCookie;

const jar = new CookieJar();
const agent = new CookieAgent({ cookies: { jar, }, });

jar.setCookie('cf_clearance=4iV5fuQ0.prnWCqcZ.kqLFV6hp.QQ5y_lRkEw97N3ZA-1653929202-0-150', 'https://nhentai.net/');

const api = new API({ agent, });

api.getBook(177013).then(book => {
	console.log(book.title.pretty);
});
To get cf_clearance cookie:

Get module User-Agent:

Run node -p "(new (require('nhentai-api').API)()).getBook(-1).catch(e => console.log(e.httpResponse.req.getHeaders()['user-agent']));"

Open https://nhentai.net/api/galleries/search in browser.

Open DevTools and set User Agent to module's:

Open bottom panel if it's not present (ESC).

Under three dots (⋮) open Network conditions.

Set custom User Agent to the module's:

Screenshot

Without closing DevTools reload page by pressing to URL and pressing enter (needed to refresh headers)

Wait for Cloudflare to complete.

Read cookie from Network tab and request (make sure you're watching at request with your User-Agent, on screenshot you can see it as the last header):

Screenshot

Place cookie to code.

Is this permanent or need to be refreshed once in a while? 🤔

Zekfad · 2022-05-31T18:46:30Z

@Chaoray, sadly, it's temporary and you'll need to refresh it,
We're working on a more automated solution, when it's done (or when maybe site admins disable CF on API) I'll tell you more info.

Chaoray · 2022-06-01T12:03:18Z

OK, I think it is enough for personal usage.
I will wait for furthur infomation 😊

Tenandrobilgi · 2022-06-04T06:25:37Z

Having the same problem. Tried the solution above but still doesn't work.

Zekfad · 2022-06-05T19:37:31Z

Make sure you got right token for right User Agents, check if UA Header is the one using by your node.

sinkaroid · 2022-06-06T07:34:06Z

There is at least has 2 workarounds:

Like mentioned above, You need some tricky with cf_clearance stuff.
It is works outside this module, I've implement that raw request through phin. But it seems cookie have an expire time, so you need to set it again.
Use their IP. After doing some reversing on nhentai.net, Looks like i get the real ip behind cloudflare https://github.com/sinkaroid/jandapress/blob/master/src/utils/options.ts#L7..L9

@Zekfad one question,
So I can't arbitrary set? When i fill it with your module nhentai-api-client/3.4.3 Node.js/16.9.1 It's works

But when i just arbitrary ie: mymodule/1.0.0 Node.js/16.9.1 returns 503 aka still cloudflare protection.
It's working as expected? I can't using own module to mock them

import p from "phin";
import { CookieJar } from "tough-cookie";
import { HttpsCookieAgent } from "http-cookie-agent/http";

const jar = new CookieJar();
jar.setCookie("some cookie", "https://nhentai.net/");

async function test() {
  const res = await p({
    url: "https://nhentai.net/api/galleries/search?query=futa",
    core: {
      agent: new HttpsCookieAgent({ cookies: { jar, }, }),
    },
    "headers": {
      "User-Agent": "jandapress/1.0.5 Node.js/16.9.1" // just worked with "nhentai-api-client", cannot use my own even i've already to set it
    },
  });
  console.log(res.statusCode);
}

test();

Zekfad · 2022-06-06T09:18:06Z

@sinkaroid token is linked to the user agent, so you have to get new one for each UA you use.

sinkaroid · 2022-06-06T10:31:00Z

Thanks it's working now

Chaoray · 2022-06-07T12:11:13Z

There is at least has 2 workarounds:

Like mentioned above, You need some tricky with cf_clearance stuff.
It is works outside this module, I've implement that raw request through phin. But it seems cookie have an expire time, so you need to set it again.

Use their IP. After doing some reversing on nhentai.net, Looks like i get the real ip behind cloudflare https://github.com/sinkaroid/jandapress/blob/master/src/utils/options.ts#L7..L9

@Zekfad one question, So I can't arbitrary set? When i fill it with your module nhentai-api-client/3.4.3 Node.js/16.9.1 It's works

But when i just arbitrary ie: mymodule/1.0.0 Node.js/16.9.1 returns 503 aka still cloudflare protection. It's working as expected? I can't using own module to mock them
import p from "phin";
import { CookieJar } from "tough-cookie";
import { HttpsCookieAgent } from "http-cookie-agent/http";

const jar = new CookieJar();
jar.setCookie("some cookie", "https://nhentai.net/");

async function test() {
  const res = await p({
    url: "https://nhentai.net/api/galleries/search?query=futa",
    core: {
      agent: new HttpsCookieAgent({ cookies: { jar, }, }),
    },
    "headers": {
      "User-Agent": "jandapress/1.0.5 Node.js/16.9.1" // just worked with "nhentai-api-client", cannot use my own even i've already to set it
    },
  });
  console.log(res.statusCode);
}

test();

How to use their IP?
Send requests directly to these IP?

Edit:
I still got 503 after I checked all steps and settings.
Is this won't work on host like heroku?
But It worked on my pc when testing.....
I have no idea

Zekfad · 2022-06-07T23:53:50Z

Is this won't work on host like heroku?

Indeed it won't. I mean, token is IP bound. If you have your server you can of course start proxy, and complete challenge though proxy, to get valid token for given IP.
But that's not an option for heroku I guess.
You can try method from #25 (comment)
But I'd be against it, because it will give them a reason to change IPs of their servers if they'll get too much traffic.

Chaoray · 2022-06-08T02:10:25Z

Is this won't work on host like heroku?

Indeed it won't. I mean, token is IP bound. If you have your server you can of course start proxy, and complete challenge though proxy, to get valid token for given IP. But that's not an option for heroku I guess. You can try method from #25 (comment) But I'd be against it, because it will give them a reason to change IPs of their servers if they'll get too much traffic.

Ok¯\_(ツ)_/¯
Now the problem is how to apply ip to nhentai-api :D

const api = new API({
    hosts: {
        api: '35.186.156.165'
    },
    ssl: false
});

Like this?

Tenandrobilgi · 2022-06-08T16:00:12Z

Is this won't work on host like heroku?

Indeed it won't. I mean, token is IP bound. If you have your server you can of course start proxy, and complete challenge though proxy, to get valid token for given IP. But that's not an option for heroku I guess. You can try method from #25 (comment) But I'd be against it, because it will give them a reason to change IPs of their servers if they'll get too much traffic.

Oh, so that's why it wasn't working.
But how would I go on about using their IP to get through Cloudflare?

Zekfad · 2022-06-08T16:07:52Z

@Chaoray, just IP, don't specify URI. Also, disable SSL.
@Tenandrobilgi, you can use proxy I think, if it's not against the TOS, Chromium based browsers has command-line for that.

Chaoray · 2022-06-09T11:13:08Z

Using site IPs is perfect but getRandomBook() lost its function. :( sad

Chaoray · 2022-08-15T09:49:53Z

Cloudflare blocked direct ip access.

And, error logs:

APIError: connect ETIMEDOUT 104.21.66.123:80
    at Function.absorb (/app/node_modules/nhentai-api/dist/cjs/bundle.cjs:471:66)
    at ClientRequest.<anonymous> (/app/node_modules/nhentai-api/dist/cjs/bundle.cjs:523:647)
    at ClientRequest.emit (node:events:527:28)
    at Socket.socketErrorListener (node:_http_client:454:9)
    at Socket.emit (node:events:527:28)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  originalError: Error: connect ETIMEDOUT 104.21.66.123:80
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1187:16) {
    errno: -110,
    code: 'ETIMEDOUT',
    syscall: 'connect',
    address: '104.21.66.123',
    port: 80
  },
  httpResponse: null
}

Hary1723 · 2022-08-28T11:38:53Z

Cloudflare blocked direct ip access. And, error logs:

APIError: connect ETIMEDOUT 104.21.66.123:80
    at Function.absorb (/app/node_modules/nhentai-api/dist/cjs/bundle.cjs:471:66)
    at ClientRequest.<anonymous> (/app/node_modules/nhentai-api/dist/cjs/bundle.cjs:523:647)
    at ClientRequest.emit (node:events:527:28)
    at Socket.socketErrorListener (node:_http_client:454:9)
    at Socket.emit (node:events:527:28)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  originalError: Error: connect ETIMEDOUT 104.21.66.123:80
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1187:16) {
    errno: -110,
    code: 'ETIMEDOUT',
    syscall: 'connect',
    address: '104.21.66.123',
    port: 80
  },
  httpResponse: null
}

~~What is that IP address?~~ Nevermind

Hary1723 · 2022-09-10T02:16:13Z

@Zekfad How would I use IP instead of the default URL? I tried from this but kept getting errors

const api = new API({
    hosts: {
        api: '35.186.156.165'
    },
    ssl: false
});

Zekfad · 2022-09-10T10:57:57Z

I guess it's no longer possible to bypass it this way, as both IPs I know doesn't work.

Hary1723 · 2022-09-10T12:36:59Z

The IP http://138.2.77.198:3002 works on the browser though. at least to me

haine4 · 2022-09-10T14:28:45Z

Current workaround: You'll need tough-cookie and http-cookie-agent for this.

// index.mjs
import { API, TagTypes, } from 'nhentai-api';
import { CookieJar } from 'tough-cookie';

import _httpCookie from 'http-cookie-agent/http';
const { HttpsCookieAgent: CookieAgent } = _httpCookie;

const jar = new CookieJar();
const agent = new CookieAgent({ cookies: { jar, }, });

jar.setCookie('cf_clearance=4iV5fuQ0.prnWCqcZ.kqLFV6hp.QQ5y_lRkEw97N3ZA-1653929202-0-150', 'https://nhentai.net/');

const api = new API({ agent, });

api.getBook(177013).then(book => {
	console.log(book.title.pretty);
});

To get cf_clearance cookie:

* Get module User-Agent:
  
  * Run `node -p "(new (require('nhentai-api').API)()).getBook(-1).catch(e => console.log(e.httpResponse.req.getHeaders()['user-agent']));"`

* Open https://nhentai.net/api/galleries/search in browser.

* Open DevTools and set User Agent to module's:
  
  * Open bottom panel if it's not present (ESC).
  * Under three dots (`⋮`) open **Network conditions**.
  * Set custom User Agent to the module's:
    
    Screenshot
    ![image](https://user-images.githubusercontent.com/8970959/171036425-34d04def-c785-49fa-9ea7-ee46257778be.png)

* Without closing DevTools reload page by pressing to URL and pressing enter (needed to refresh headers)

* Wait for Cloudflare to complete.

* Read cookie from **Network** tab and request (make sure you're watching at request with your User-Agent, on screenshot you can see it as the last header):
  
  Screenshot
  ![image](https://user-images.githubusercontent.com/8970959/171037214-d4767f8d-4d8f-40d3-bd94-84b6c47cc4e8.png)

* Place cookie to code.

Hey It's working on my local, but how to use this with repl / railway / or heroku?

haine4 · 2022-09-10T14:34:23Z

The IP http://138.2.77.198:3002 works on the browser though. at least to me

It's the nhentai address? how did you know?

Hary1723 · 2022-09-10T14:41:16Z

Hey It's working on my local, but how to use this with repl / railway / or heroku?

Same question here, hence I thought using the IP might work.

It's the nhentai address? how did you know?

https://github.com/sinkaroid/jandapress/blob/master/src/utils/options.ts#L10

Uryaaa · 2022-09-16T06:35:48Z

The IP http://138.2.77.198:

Now how to use this with nhentai-api

Chaoray · 2022-09-23T11:42:23Z

No, you can't. Nhentai blocked direct ip access, which was the previous method we were using before it was blocked.
Or you can do one thing that coverts a book number to thumbnail urls and image urls and basically just let them load from user client.

lewisakura · 2022-10-15T22:33:39Z

Nhentai blocked direct ip access

@Chaoray this is incorrect. The direct IP access error comes from the fact that the IP the person attempted to use was a Cloudflare network IP, which naturally disallows direct access (otherwise how would they know what you're trying to connect to?). If you navigate to the IP and port, with http (not https!), you can in fact get it to load just fine. It looks broken because all of the CSS isn't there, but other than that it is fully functional, API included.

With a bit of patching, you can get the library to work fine with the IP address:

diff --git a/node_modules/nhentai-api/dist/cjs/bundle.cjs b/node_modules/nhentai-api/dist/cjs/bundle.cjs
index 46ad4e5..840d3b3 100644
--- a/node_modules/nhentai-api/dist/cjs/bundle.cjs
+++ b/node_modules/nhentai-api/dist/cjs/bundle.cjs
@@ -507,7 +507,7 @@ class API{
    * Applies provided options on top of defaults.
    * @param {?nHentaiOptions} [options={}] Options to apply.
    */
-constructor(options={}){_defineProperty(this,"hosts",void 0),_defineProperty(this,"ssl",void 0),_defineProperty(this,"agent",void 0);let params=function processOptions({hosts:{api:api="nhentai.net",images:images="i.nhentai.net",thumbs:thumbs="t.nhentai.net"}={},ssl:ssl=!0,agent:agent=null}={}){return agent||(agent=ssl?https.Agent:http.Agent),"Function"===agent.constructor.name&&(agent=new agent),{hosts:{api:api,images:images,thumbs:thumbs},ssl:ssl,agent:agent}}(options);Object.assign(this,params)}
+constructor(options={}){_defineProperty(this,"hosts",void 0),_defineProperty(this,"ssl",void 0),_defineProperty(this,"agent",void 0);let params=function processOptions({hosts:{api:api="138.2.77.198",images:images="i.nhentai.net",thumbs:thumbs="t.nhentai.net"}={},ssl:ssl=!0,agent:agent=null}={}){return agent||(agent=ssl?https.Agent:http.Agent),"Function"===agent.constructor.name&&(agent=new agent),{hosts:{api:api,images:images,thumbs:thumbs},ssl:ssl,agent:agent}}(options);Object.assign(this,params)}
 /**
    * Get http(s) module depending on `options.ssl`.
    * @type {https|http}
@@ -518,7 +518,7 @@ constructor(options={}){_defineProperty(this,"hosts",void 0),_defineProperty(thi
    * @param {string} options.host Host.
    * @param {string} options.path Path.
    * @returns {Promise<object>} Parsed JSON.
-   */request(options){let{net:net,agent:agent}=this;return new Promise(((resolve,reject)=>{Object.assign(options,{agent:agent,headers:{"User-Agent":`nhentai-api-client/3.4.3 Node.js/${process.versions.node}`}}),net.get(options,(_response=>{const
+   */request(options){let net=(options.host==="138.2.77.198"?http__default.default:this.net),agent=(options.host==="138.2.77.198"?new http.Agent():this.agent);return new Promise(((resolve,reject)=>{Object.assign(options,{agent:agent,headers:{"User-Agent":`nhentai-api-client/3.4.3 Node.js/${process.versions.node}`}}),net.get({...options,port:options.host==="138.2.77.198"?3002:undefined},(_response=>{const
 /** @type {IncomingMessage}*/
 response=_response,{statusCode:statusCode}=response,contentType=response.headers["content-type"];let error;if(200!==statusCode?error=new Error(`Request failed with status code ${statusCode}`):/^application\/json/.test(contentType)||(error=new Error(`Invalid content-type - expected application/json but received ${contentType}`)),error)return response.resume(),void reject(APIError.absorb(error,response));response.setEncoding("utf8");let rawData="";response.on("data",(chunk=>rawData+=chunk)),response.on("end",(()=>{try{resolve(JSON.parse(rawData))}catch(error){reject(APIError.absorb(error,response))}}))})).on("error",(error=>reject(APIError.absorb(error))))}))}
 /**

(this can be thrown into a patch-package patches folder with the name nhentai-api+3.4.3.patch if you want this to work now)

Obviously this patch is very.. hacky: it doesn't support custom agents (if you use those) and it recreates the HTTP agent every time, but it does indeed work. First-party support shouldn't be too terrible and I'll probably contribute it myself when I get around to it.

Zekfad · 2022-10-16T14:37:34Z

Issue regarding port number support can be tracked in #26 now.

Chaoray · 2022-10-17T14:16:40Z

@LewisTehMinerz Can you tell me how to get the real ip behind cloudflare?

lewisakura · 2022-10-17T14:24:52Z

@LewisTehMinerz Can you tell me how to get the real ip behind cloudflare?

You can't. It's just that nhentai's IP address was already known and appears to have been public for ages. I got that IP address from the previous posts here.

Chaoray · 2022-10-17T14:42:02Z

Ok, I tried your solution but it seemed not working to me :P
Kept getting code 400.
WHY nhentai have to do things like these???

soujiokita · 2023-02-08T22:15:38Z

Current workaround: You'll need tough-cookie and http-cookie-agent for this.

// index.mjs
import { API, TagTypes, } from 'nhentai-api';
import { CookieJar } from 'tough-cookie';

import _httpCookie from 'http-cookie-agent/http';
const { HttpsCookieAgent: CookieAgent } = _httpCookie;

const jar = new CookieJar();
const agent = new CookieAgent({ cookies: { jar, }, });

jar.setCookie('cf_clearance=4iV5fuQ0.prnWCqcZ.kqLFV6hp.QQ5y_lRkEw97N3ZA-1653929202-0-150', 'https://nhentai.net/');

const api = new API({ agent, });

api.getBook(177013).then(book => {
	console.log(book.title.pretty);
});

To get cf_clearance cookie:

* Get module User-Agent:
  
  * Run `node -p "(new (require('nhentai-api').API)()).getBook(-1).catch(e => console.log(e.httpResponse.req.getHeaders()['user-agent']));"`

* Open https://nhentai.net/api/galleries/search in browser.

* Open DevTools and set User Agent to module's:
  
  * Open bottom panel if it's not present (ESC).
  * Under three dots (`⋮`) open **Network conditions**.
  * Set custom User Agent to the module's:
    
    Screenshot
    ![image](https://user-images.githubusercontent.com/8970959/171036425-34d04def-c785-49fa-9ea7-ee46257778be.png)

* Without closing DevTools reload page by pressing to URL and pressing enter (needed to refresh headers)

* Wait for Cloudflare to complete.

* Read cookie from **Network** tab and request (make sure you're watching at request with your User-Agent, on screenshot you can see it as the last header):
  
  Screenshot
  ![image](https://user-images.githubusercontent.com/8970959/171037214-d4767f8d-4d8f-40d3-bd94-84b6c47cc4e8.png)

* Place cookie to code.

How long cf_clearance will expire?

Zekfad · 2023-02-09T10:25:43Z

About few hours, you can see cookie expiration date.

soujiokita · 2023-02-09T17:54:18Z

@Zekfad where is the actual i can view the cookie expiration date, it's on chrome browser or where,

I tested the workaround but with some interval requests, has over ±22 hours and cf_clearance still works

Zekfad · 2023-02-09T19:18:17Z

@soujiokita DevTools -> Application -> Cookies, there is the Expiration date.

Uryaaa · 2023-02-11T04:36:14Z

The IP http://138.2.77.198:3002 works on the browser though. at least to me

they enabled cf on this IP yesterday for a few hours

Zekfad changed the title ~~Getting 503 responses most likely due to cloudflare~~ Cloudflare enabled on API endpoints May 24, 2022

Zekfad mentioned this issue Dec 21, 2022

... Zekfad/nhentai_dart#5

Closed

Zekfad added the help wanted Extra attention is needed label Jan 28, 2023

Cloudflare enabled on API endpoints #25

Cloudflare enabled on API endpoints #25

Comments

cerinoligutom commented May 22, 2022

Zekfad commented May 24, 2022

Chaoray commented May 29, 2022

Zekfad commented May 30, 2022 • edited Loading

Chaoray commented May 31, 2022

Zekfad commented May 31, 2022

Chaoray commented Jun 1, 2022

Tenandrobilgi commented Jun 4, 2022 • edited Loading

Zekfad commented Jun 5, 2022

sinkaroid commented Jun 6, 2022 • edited Loading

Zekfad commented Jun 6, 2022

sinkaroid commented Jun 6, 2022

Chaoray commented Jun 7, 2022 • edited Loading

Zekfad commented Jun 7, 2022

Chaoray commented Jun 8, 2022 • edited Loading

Tenandrobilgi commented Jun 8, 2022 • edited Loading

Zekfad commented Jun 8, 2022

Chaoray commented Jun 9, 2022

Chaoray commented Aug 15, 2022 • edited Loading

Hary1723 commented Aug 28, 2022 • edited Loading

Hary1723 commented Sep 10, 2022

Zekfad commented Sep 10, 2022

Hary1723 commented Sep 10, 2022 • edited Loading

haine4 commented Sep 10, 2022

haine4 commented Sep 10, 2022

Hary1723 commented Sep 10, 2022

Uryaaa commented Sep 16, 2022

Chaoray commented Sep 23, 2022 • edited Loading

lewisakura commented Oct 15, 2022

Zekfad commented Oct 16, 2022

Chaoray commented Oct 17, 2022

lewisakura commented Oct 17, 2022

Chaoray commented Oct 17, 2022

soujiokita commented Feb 8, 2023

Zekfad commented Feb 9, 2023

soujiokita commented Feb 9, 2023

Zekfad commented Feb 9, 2023

Uryaaa commented Feb 11, 2023

Zekfad commented May 30, 2022 •

edited

Loading

Tenandrobilgi commented Jun 4, 2022 •

edited

Loading

sinkaroid commented Jun 6, 2022 •

edited

Loading

Chaoray commented Jun 7, 2022 •

edited

Loading

Chaoray commented Jun 8, 2022 •

edited

Loading

Tenandrobilgi commented Jun 8, 2022 •

edited

Loading

Chaoray commented Aug 15, 2022 •

edited

Loading

Hary1723 commented Aug 28, 2022 •

edited

Loading

Hary1723 commented Sep 10, 2022 •

edited

Loading

Chaoray commented Sep 23, 2022 •

edited

Loading