Skip to content

Commit

Permalink
AG-16687 add xml-prune scriptlet
Browse files Browse the repository at this point in the history 
Squashed commit of the following:

commit 18a5a61
Merge: 93efce6 0976bb9
Author: Slava Leleka <v.leleka@adguard.com>
Date:   Tue Oct 18 00:57:30 2022 +0300

    Merge branch 'feature/AG-16687_01' of ssh://bit.adguard.com:7999/adguard-filters/scriptlets into feature/AG-16687_01

commit 93efce6
Merge: 10ee63d 66b417a
Author: Slava Leleka <v.leleka@adguard.com>
Date:   Tue Oct 18 00:57:12 2022 +0300

    merge release/v1.7 into feature/AG-16687_01, resolve conflicts

commit 0976bb9
Author: Adam <adam@adguard.com>
Date:   Fri Oct 14 18:47:47 2022 +0200

    Remove unnecessary variable

commit 10ee63d
Author: Adam <adam@adguard.com>
Date:   Fri Oct 14 14:51:54 2022 +0200

    Return original value if response is not pruned
    Fix tests

commit 0053d0c
Author: Adam <adam@adguard.com>
Date:   Fri Oct 14 11:40:53 2022 +0200

    Remove unnecessary conditional statement

commit 930dd6a
Merge: fe26660 cfa9570
Author: Adam <adam@adguard.com>
Date:   Thu Oct 13 12:41:31 2022 +0200

    Merge branch 'release/v1.7' into feature/AG-16687_01

commit fe26660
Author: Adam <adam@adguard.com>
Date:   Thu Oct 13 12:40:41 2022 +0200

    Add common constant GET_METHOD for tests
    Remove unnecessary MPD_PATH

commit 63c4bef
Author: Adam <adam@adguard.com>
Date:   Wed Oct 12 14:03:21 2022 +0200

    Rename url to urlMatchRegexp

commit e43bd18
Author: Adam <adam@adguard.com>
Date:   Wed Oct 12 13:46:21 2022 +0200

    Avoid regexp
    Improve log
    Add a note about usage without propsToMatch
    Change realFetch to nativeFetch
    Do not reassign input variable
    Add validity of xmlDoc
    Get rid of try...catch
    Remove unnecessary spaces in test file
    Fix endsWith function

commit 8185ae3
Author: Adam <adam@adguard.com>
Date:   Mon Oct 10 14:33:28 2022 +0200

    Remove unnecessary hit function
    Rename shouldLog to shouldPruneResponse

commit cf1f380
Author: Adam <adam@adguard.com>
Date:   Fri Oct 7 17:47:07 2022 +0200

    Update description

commit f358b8a
Author: Adam <adam@adguard.com>
Date:   Fri Oct 7 14:36:48 2022 +0200

    Rename checkIfXML functin to isXML

commit c197e0a
Author: Adam <adam@adguard.com>
Date:   Fri Oct 7 14:28:45 2022 +0200

    Add ability to log response in a browser console
    Add tests for logging
    Use assert.notOk
    Improve description
    Remove unnecessary hit function
    Simplify test file
    Add removeEventListener() method
    Improve function name (pruneXML)
    Add checkIfXML function and comment to it
    Check if Reflect is supported
    Add eslint-enable max-len
    Use indexOf() instead of includes() in tests
    Add more aliases

commit 66b76c0
Author: Adam <adam@adguard.com>
Date:   Tue Oct 4 10:19:31 2022 +0200

    Add new scriptlet - `xml-prune`
  • Loading branch information
@AdamWr
AdamWr committed Oct 18, 2022
1 parent 66b417a commit e1489ee
Show file tree
Hide file tree
Showing 7 changed files with 622 additions and 1 deletion.
2 changes: 1 addition & 1 deletion src/helpers/string-utils.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ export const startsWith = (str, prefix) => {
export const endsWith = (str, ending) => {
// if str === '', (str && false) will return ''
// that's why it has to be !!str
return !!str && str.indexOf(ending) === str.length - ending.length;
return !!str && str.lastIndexOf(ending) === str.length - ending.length;
};

export const substringAfter = (str, separator) => {
Expand Down
1 change: 1 addition & 0 deletions src/scriptlets/scriptlets-list.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,4 @@ export * from './close-window';
export * from './prevent-refresh';
export * from './prevent-element-src-loading';
export * from './no-topics';
export * from './xml-prune';
217 changes: 217 additions & 0 deletions src/scriptlets/xml-prune.js
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,217 @@
import {
hit,
toRegExp,
startsWith,
endsWith,
} from '../helpers/index';

/* eslint-disable max-len */
/**
* @scriptlet xml-prune
*
* @description
* Removes an element from the specified XML.
*
*
* **Syntax**
* ```
* example.org#%#//scriptlet('xml-prune'[, propsToMatch[, optionalProp[, urlToMatch]]])
* ```
*
* - `propsToMatch` - optional, selector of elements which will be removed from XML
* - `optionalProp` - optional, selector of elements that must occur in XML document
* - `urlToMatch` - optional, string or regular expression for matching the request's URL
* > Usage with no arguments will log response payload and URL to browser console;
* which is useful for debugging but prohibited for production filter lists.
*
* **Examples**
* 1. Remove `Period` tag whose `id` contains `-ad-` from all requests
* ```
* example.org#%#//scriptlet('xml-prune', 'Period[id*="-ad-"]')
* ```
*
* 2. Remove `Period` tag whose `id` contains `-ad-`, only if XML contains `SegmentTemplate`
* ```
* example.org#%#//scriptlet('xml-prune', 'Period[id*="-ad-"]', 'SegmentTemplate')
* ```
*
* 3. Remove `Period` tag whose `id` contains `-ad-`, only if request's URL contains `.mpd`
* ```
* example.org#%#//scriptlet('xml-prune', 'Period[id*="-ad-"]', '', '.mpd')
* ```
*
* 4. Call with no arguments will log response payload and URL at the console
* ```
* example.org#%#//scriptlet('xml-prune')
* ```
*
* 5. Call with only `urlToMatch` argument will log response payload and URL only for the matched URL
* ```
* example.org#%#//scriptlet('xml-prune', '', '', '.mpd')
* ```
*/
/* eslint-enable max-len */

export function xmlPrune(source, propsToRemove, optionalProp = '', urlToMatch) {
// do nothing if browser does not support Reflect, fetch or Proxy (e.g. Internet Explorer)
// https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/fetch
// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy
// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Reflect
if (typeof Reflect === 'undefined'
|| typeof fetch === 'undefined'
|| typeof Proxy === 'undefined'
|| typeof Response === 'undefined') {
return;
}

let shouldPruneResponse = true;
// eslint-disable-next-line no-console
const log = console.log.bind(console);
if (!propsToRemove) {
// If "propsToRemove" is not defined, then response shouldn't be pruned
// but it should be logged in browser console
shouldPruneResponse = false;
}

const urlMatchRegexp = toRegExp(urlToMatch);

const isXML = (text) => {
// Check if "text" starts with "<" and check if it ends with ">"
// If so, then it might be an XML file and should be pruned or logged
const trimedText = text.trim();
if (startsWith(trimedText, '<') && endsWith(trimedText, '>')) {
return true;
}
return false;
};

const pruneXML = (text) => {
if (!isXML(text)) {
shouldPruneResponse = false;
return text;
}
const xmlParser = new DOMParser();
const xmlDoc = xmlParser.parseFromString(text, 'text/xml');
const errorNode = xmlDoc.querySelector('parsererror');
if (errorNode) {
return text;
}
if (optionalProp !== '' && xmlDoc.querySelector(optionalProp) === null) {
shouldPruneResponse = false;
return text;
}
const elems = xmlDoc.querySelectorAll(propsToRemove);
if (!elems.length) {
shouldPruneResponse = false;
return text;
}
elems.forEach((elem) => {
elem.remove();
});
const serializer = new XMLSerializer();
text = serializer.serializeToString(xmlDoc);
return text;
};

const xhrWrapper = (target, thisArg, args) => {
const xhrURL = args[1];
if (typeof xhrURL !== 'string' || xhrURL.length === 0) {
return Reflect.apply(target, thisArg, args);
}
if (urlMatchRegexp.test(xhrURL)) {
thisArg.addEventListener('readystatechange', function pruneResponse() {
if (thisArg.readyState === 4) {
const { response } = thisArg;
thisArg.removeEventListener('readystatechange', pruneResponse);
if (!shouldPruneResponse) {
if (isXML(response)) {
log(`XMLHttpRequest.open() URL: ${xhrURL}\nresponse: ${response}`);
}
} else {
const prunedResponseContent = pruneXML(response);
if (shouldPruneResponse) {
Object.defineProperty(thisArg, 'response', {
value: prunedResponseContent,
});
Object.defineProperty(thisArg, 'responseText', {
value: prunedResponseContent,
});
hit(source);
}
// In case if response shouldn't be pruned
// pruneXML sets shouldPruneResponse to false
// so it's necessary to set it to true again
// otherwise response will be only logged
shouldPruneResponse = true;
}
}
});
}
return Reflect.apply(target, thisArg, args);
};

const xhrHandler = {
apply: xhrWrapper,
};
// eslint-disable-next-line max-len
window.XMLHttpRequest.prototype.open = new Proxy(window.XMLHttpRequest.prototype.open, xhrHandler);

// eslint-disable-next-line compat/compat
const nativeFetch = window.fetch;

const fetchWrapper = (target, thisArg, args) => {
const fetchURL = args[0];
if (typeof fetchURL !== 'string' || fetchURL.length === 0) {
return Reflect.apply(target, thisArg, args);
}
if (urlMatchRegexp.test(fetchURL)) {
return nativeFetch.apply(this, args).then((response) => {
return response.text().then((text) => {
if (!shouldPruneResponse) {
if (isXML(text)) {
log(`fetch URL: ${fetchURL}\nresponse text: ${text}`);
}
return Reflect.apply(target, thisArg, args);
}
const prunedText = pruneXML(text);
if (shouldPruneResponse) {
hit(source);
return new Response(prunedText, {
status: response.status,
statusText: response.statusText,
headers: response.headers,
});
}
// In case if response shouldn't be pruned
// pruneXML sets shouldPruneResponse to false
// so it's necessary to set it to true again
// otherwise response will be only logged
shouldPruneResponse = true;
return Reflect.apply(target, thisArg, args);
});
});
}
return Reflect.apply(target, thisArg, args);
};

const fetchHandler = {
apply: fetchWrapper,
};
// eslint-disable-next-line compat/compat
window.fetch = new Proxy(window.fetch, fetchHandler);
}

xmlPrune.names = [
'xml-prune',
// aliases are needed for matching the related scriptlet converted into our syntax
'xml-prune.js',
'ubo-xml-prune.js',
'ubo-xml-prune',
];

xmlPrune.injections = [
hit,
toRegExp,
startsWith,
endsWith,
];
1 change: 1 addition & 0 deletions tests/scriptlets/index.test.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,5 @@ import './close-window.test';
import './prevent-refresh.test';
import './prevent-element-src-loading.test';
import './no-topics.test';
import './xml-prune.test';
import './trusted-click-element.test';
32 changes: 32 additions & 0 deletions tests/scriptlets/test-files/manifestMPD.mpd
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
<?xml version="1.0" encoding="UTF-8"?>
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:_XMLSchema-instance="http://www.w3.org/2001/XMLSchema-instance" _XMLSchema-instance:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-live:2011" type="static" mediaPresentationDuration="PT49M2.272666664S" minBufferTime="PT2S">
<BaseURL>https://vod-gcs-cedexis.cbsaavideo.com/intl_vms/2017/02/17/879659075884/609941_cenc_precon_dash/</BaseURL>
<Period id="pre-roll-1-ad-1" duration="PT20.02S">
<BaseURL>https://dai.google.com/segments/redirect/c/</BaseURL>
</Period>
<Period id="0" duration="PT21M23.282S">
<AdaptationSet id="0" lang="en" contentType="text" segmentAlignment="true">
<Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
</AdaptationSet>
</Period>
<Period id="1" duration="PT12M48.768S">
<AdaptationSet id="0" lang="en" contentType="text" segmentAlignment="true">
<Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
</AdaptationSet>
</Period>
<Period id="2" duration="PT6M26.385999999S">
<AdaptationSet id="0" lang="en" contentType="text" segmentAlignment="true">
<Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
</AdaptationSet>
</Period>
<Period id="3" duration="PT7M12.431999999S">
<AdaptationSet id="0" lang="en" contentType="text" segmentAlignment="true">
<Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
</AdaptationSet>
</Period>
<Period id="4" duration="PT51.384666666S">
<AdaptationSet id="0" lang="en" contentType="text" segmentAlignment="true">
<Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
</AdaptationSet>
</Period>
</MPD>
Loading

0 comments on commit e1489ee

Please sign in to comment.