Wiki source code of Migration progress
Last modified by Eleni Cojocariu on 2026/10/12 15:10
Show last authors
| author | version | line-number | content |
|---|---|---|---|
| 1 | {{warning}} | ||
| 2 | WIP | ||
| 3 | {{/warning}} | ||
| 4 | |||
| 5 | {{toc/}} | ||
| 6 | |||
| 7 | The numbers below are computed when this page is displayed, by querying the search index. Nothing is stored. | ||
| 8 | |||
| 9 | A legacy page counts as **done** when it carries the ##done## tag. It counts as **started** when it holds a [[MigratedPageClass>>XWikiOrg.documentation-scripts-queries.MigratedPageClass]] object listing its replacements but has not been tagged yet. It counts as **unconvertable** when it carries the ##unconvertable## tag: its content is deprecated and has no place in the new documentation, so there is nothing to convert and nothing to map it to. Both tags take a page out of the //Left// column, which is the work that is still ahead. | ||
| 10 | |||
| 11 | == Legacy Pages on the Main wiki (this wiki) == | ||
| 12 | |||
| 13 | {{velocity}} | ||
| 14 | #set ($Q = '"') | ||
| 15 | #set ($mappingClass = 'XWikiOrg.documentation-scripts-queries.MigratedPageClass') | ||
| 16 | #set ($detailsPage = 'XWikiOrg.documentation-scripts-queries.migration-progress.details.WebHome') | ||
| 17 | #set ($trees = []) | ||
| 18 | #set ($discard = $trees.add({'key': 'user', 'space': 'Documentation.UserGuide'})) | ||
| 19 | #set ($discard = $trees.add({'key': 'admin', 'space': 'Documentation.AdminGuide'})) | ||
| 20 | #set ($discard = $trees.add({'key': 'dev', 'space': 'Documentation.DevGuide'})) | ||
| 21 | #set ($totalPages = 0) | ||
| 22 | #set ($totalDone = 0) | ||
| 23 | #set ($totalUnconvertable = 0) | ||
| 24 | #set ($totalStarted = 0) | ||
| 25 | #set ($totalLeft = 0) | ||
| 26 | (% class="table" %) | ||
| 27 | |=Tree|=Pages|=Done|=Unconvertable|=In progress|=Left|=Page by page | ||
| 28 | #foreach ($tree in $trees) | ||
| 29 | #set ($base = "type:DOCUMENT AND wiki:xwiki AND hidden:false AND space_prefix:${Q}${tree.space}${Q} AND -name:WebPreferences") | ||
| 30 | #set ($pages = $services.query.createQuery($base, 'solr').setLimit(0).execute().get(0).results.numFound) | ||
| 31 | #set ($done = $services.query.createQuery("${base} AND property.XWiki.TagClass.tags:done", 'solr').setLimit(0).execute().get(0).results.numFound) | ||
| 32 | #set ($unconvertable = $services.query.createQuery("${base} AND property.XWiki.TagClass.tags:unconvertable", 'solr').setLimit(0).execute().get(0).results.numFound) | ||
| 33 | #set ($started = $services.query.createQuery("${base} AND class:${Q}${mappingClass}${Q} AND -property.XWiki.TagClass.tags:done AND -property.XWiki.TagClass.tags:unconvertable", 'solr').setLimit(0).execute().get(0).results.numFound) | ||
| 34 | ## Left is queried rather than subtracted: nothing stops a page from carrying both tags, and | ||
| 35 | ## subtracting the two counts would then take it out of the column twice. | ||
| 36 | #set ($left = $services.query.createQuery("${base} AND -property.XWiki.TagClass.tags:done AND -property.XWiki.TagClass.tags:unconvertable", 'solr').setLimit(0).execute().get(0).results.numFound) | ||
| 37 | #set ($totalPages = $totalPages + $pages) | ||
| 38 | #set ($totalDone = $totalDone + $done) | ||
| 39 | #set ($totalUnconvertable = $totalUnconvertable + $unconvertable) | ||
| 40 | #set ($totalStarted = $totalStarted + $started) | ||
| 41 | #set ($totalLeft = $totalLeft + $left) | ||
| 42 | #set ($detailsURL = $xwiki.getURL($detailsPage, 'view', "audience=$tree.key")) | ||
| 43 | |[[$tree.space>>doc:${tree.space}.WebHome]]|$pages|$done|$unconvertable|$started|$left|[[View detailed info>>path:$detailsURL]] | ||
| 44 | #end | ||
| 45 | |**Total**|**$totalPages**|**$totalDone**|**$totalUnconvertable**|**$totalStarted**|**$totalLeft**| | ||
| 46 | {{/velocity}} | ||
| 47 | |||
| 48 | == Pages on extension.xwiki.org == | ||
| 49 | |||
| 50 | An extension page needs no tag: its ##website## field is filled when the page has been converted, and points at the extension's generated documentation listing (##/bin/view/DocApp/Code/ExtensionLD?id=...##). | ||
| 51 | |||
| 52 | Pages are not extensions, though. A page can carry an empty ##id##, and the same ##id## can be carried by several pages, so what a ratio can honestly be computed against is the number of //distinct ids//: that is the set of things there is to document. Both are counted below, the distinct ids by faceting the index on the id field. | ||
| 53 | |||
| 54 | The last row counts a different population, and one the three above miss entirely: | ||
| 55 | |||
| 56 | * a **project page** carries an ##ExtensionCode.ProjectClass## object and //no// ##ExtensionClass## one - it groups the modules of a single project under one page (for instance [[Extension.Documentation>>extensions:Extension.Documentation.WebHome]]), so it is not an extension page and none of the rows above sees it; | ||
| 57 | * a **pom page** carries an ##ExtensionCode.ExtensionProxyClass## object: its metadata is imported from a Maven repository rather than typed on the wiki, so its ##website## comes from the pom. | ||
| 58 | |||
| 59 | The two sets overlap, so the row counts their union and not their sum. | ||
| 60 | |||
| 61 | {{velocity}} | ||
| 62 | #set ($Q = '"') | ||
| 63 | #set ($idField = 'property.ExtensionCode.ExtensionClass.id_string') | ||
| 64 | #set ($extBase = "type:DOCUMENT AND wiki:extensions AND hidden:false AND class:${Q}ExtensionCode.ExtensionClass${Q}") | ||
| 65 | #set ($detailsPage = 'XWikiOrg.documentation-scripts-queries.migration-progress.extensions-details.WebHome') | ||
| 66 | #set ($all = $services.query.createQuery($extBase, 'solr').setLimit(0).execute().get(0).results.numFound) | ||
| 67 | #set ($withId = $services.query.createQuery("${extBase} AND ${idField}:?*", 'solr').setLimit(0).execute().get(0).results.numFound) | ||
| 68 | ## "Done" is a filled website field. It is a TextArea, so it has no '_string' twin and its real field | ||
| 69 | ## carries a locale suffix -- but the bare name is the one that resolves in a query, and ':?*' (one | ||
| 70 | ## character, then anything) is what distinguishes a filled value from an empty one: ':[* TO *]', | ||
| 71 | ## ':*' and '-:""' all match every page, empty ones included. | ||
| 72 | #set ($websiteField = 'property.ExtensionCode.ExtensionClass.website') | ||
| 73 | #set ($allDone = $services.query.createQuery("${extBase} AND ${websiteField}:?*", 'solr').setLimit(0).execute().get(0).results.numFound) | ||
| 74 | #set ($withIdDone = $services.query.createQuery("${extBase} AND ${idField}:?* AND ${websiteField}:?*", 'solr').setLimit(0).execute().get(0).results.numFound) | ||
| 75 | #set ($noId = $all - $withId) | ||
| 76 | #set ($noIdDone = $allDone - $withIdDone) | ||
| 77 | ## Project pages and pom pages. A project page carries no ExtensionClass object at all, so it is in | ||
| 78 | ## none of the three scopes above; a pom page is an extension page whose fields are imported from a | ||
| 79 | ## Maven repository. 19 pages are both, hence the OR -- adding the two counts would double-count them. | ||
| 80 | ## A project page's own "done" signal is its ProjectClass website field, not the ExtensionClass one. | ||
| 81 | #set ($projectWebsiteField = 'property.ExtensionCode.ProjectClass.website') | ||
| 82 | #set ($pomBase = "type:DOCUMENT AND wiki:extensions AND hidden:false AND (class:${Q}ExtensionCode.ProjectClass${Q} OR class:${Q}ExtensionCode.ExtensionProxyClass${Q})") | ||
| 83 | #set ($pomAll = $services.query.createQuery($pomBase, 'solr').setLimit(0).execute().get(0).results.numFound) | ||
| 84 | #set ($pomDone = $services.query.createQuery("${pomBase} AND (${projectWebsiteField}:?* OR ${websiteField}:?*)", 'solr').setLimit(0).execute().get(0).results.numFound) | ||
| 85 | #set ($rows = []) | ||
| 86 | #set ($discard = $rows.add({'label': 'All pages on extensions.xwiki.org', 'scope': '', 'number': $all, 'done': $allDone})) | ||
| 87 | #set ($discard = $rows.add({'label': 'Pages without id', 'scope': 'noid', 'number': $noId, 'done': $noIdDone})) | ||
| 88 | #set ($discard = $rows.add({'label': 'Pages with id', 'scope': 'withid', 'number': $withId, 'done': $withIdDone})) | ||
| 89 | #set ($discard = $rows.add({'label': 'Project page or pom page', 'scope': 'project', 'number': $pomAll, 'done': $pomDone})) | ||
| 90 | (% class="table" %) | ||
| 91 | |=Scope|=Number|=Done|=View detailed | ||
| 92 | #foreach ($row in $rows) | ||
| 93 | #if ($row.scope == '') | ||
| 94 | #set ($link = '') | ||
| 95 | #else | ||
| 96 | #set ($url = $xwiki.getURL($detailsPage, 'view', "scope=$row.scope")) | ||
| 97 | #set ($link = "[[View detailed>>path:$url]]") | ||
| 98 | #end | ||
| 99 | |$row.label|$row.number|$row.done|$link | ||
| 100 | #end | ||
| 101 | {{/velocity}} | ||
| 102 | |||
| 103 | {{info}} | ||
| 104 | The scope is still an open question, not a measurement problem: most contrib extensions are not in the scope of the migration, so the second row stays low whatever happens, and the first row under-reports the work done because part of what has been converted so far is contrib, not platform. The distinct-id columns are the denominators to use once the target set is decided. | ||
| 105 | {{/info}} | ||
| 106 | |||
| 107 | == Pages on rendering.xwiki.org == | ||
| 108 | |||
| 109 | [[rendering.xwiki.org>>https://rendering.xwiki.org/xwiki/bin/view/Main/]] is a wiki of the same farm, so the same index answers for it. The number below is a number of //pages//: hidden pages and per-space preference pages are left out, and so are translations - the index holds one document per locale, which on this wiki is the difference between 119 documents and 66 pages (##Main.WebHome## alone is translated 19 times). No page there carries a ##done## tag or a mapping object yet, so there is nothing to count as done. | ||
| 110 | |||
| 111 | {{velocity}} | ||
| 112 | #set ($Q = '"') | ||
| 113 | #set ($renderingBase = "type:DOCUMENT AND wiki:rendering AND hidden:false AND -name:WebPreferences AND doclocale:${Q}${Q}") | ||
| 114 | #set ($renderingPages = $services.query.createQuery($renderingBase, 'solr').setLimit(0).execute().get(0).results.numFound) | ||
| 115 | ## Main is the only space of that wiki holding rendering documentation: the other 46 pages are the | ||
| 116 | ## flavor it was installed with (Help, Sandbox, ColorThemes, Panels, XWiki). Its detail view is the | ||
| 117 | ## page-by-page one -- same columns, and the same done/converted/redirect tests, as the legacy trees. | ||
| 118 | #set ($renderingMain = $services.query.createQuery("${renderingBase} AND space_prefix:${Q}Main${Q}", 'solr').setLimit(0).execute().get(0).results.numFound) | ||
| 119 | #set ($renderingDetails = 'XWikiOrg.documentation-scripts-queries.migration-progress.rendering-details.WebHome') | ||
| 120 | #set ($renderingURL = $xwiki.getURL($renderingDetails)) | ||
| 121 | #set ($renderingMainURL = $xwiki.getURL($renderingDetails, 'view', 'scope=main')) | ||
| 122 | (% class="table" %) | ||
| 123 | |=Scope|=Number|=View detailed | ||
| 124 | |All pages on rendering.xwiki.org|$renderingPages|[[View detailed>>path:$renderingURL]] | ||
| 125 | |Main|$renderingMain|[[View detailed>>path:$renderingMainURL]] | ||
| 126 | {{/velocity}} | ||
| 127 | |||
| 128 | == FAQ Pages == | ||
| 129 | |||
| 130 | A FAQ is a page of the [[FAQ>>doc:FAQ.WebHome]] space carrying a ##FAQCode.FAQClass## object - the set its home page lists. Pages of that space without the object are not counted here. | ||
| 131 | |||
| 132 | {{velocity}} | ||
| 133 | #set ($Q = '"') | ||
| 134 | #set ($faqBase = "type:DOCUMENT AND wiki:xwiki AND hidden:false AND space_prefix:${Q}FAQ${Q} AND -name:WebPreferences AND doclocale:${Q}${Q}") | ||
| 135 | #set ($faqEntries = $services.query.createQuery("${faqBase} AND class:${Q}FAQCode.FAQClass${Q}", 'solr').setLimit(0).execute().get(0).results.numFound) | ||
| 136 | #set ($faqDetails = 'XWikiOrg.documentation-scripts-queries.migration-progress.faq-details.WebHome') | ||
| 137 | #set ($faqURL = $xwiki.getURL($faqDetails)) | ||
| 138 | (% class="table" %) | ||
| 139 | |=Scope|=Number|=View detailed | ||
| 140 | |Pages carrying a FAQ object|$faqEntries|[[View detailed>>path:$faqURL]] | ||
| 141 | {{/velocity}} | ||
| 142 | |||
| 143 | == The queries == | ||
| 144 | |||
| 145 | They are in the source of this page. A few things they all depend on: | ||
| 146 | |||
| 147 | * **Scope by ##wiki:##.** The index covers the whole farm, so an unscoped query also counts the ##cristal## wiki, which has pages under the same space paths. That alone inflated the User Guide count from 44 to 54. | ||
| 148 | * **Exclude hidden pages** with ##hidden:false##. They are technical pages, not documentation, and they are numerous: 12 in the User Guide, 39 in the Admin Guide, 126 in the Dev Guide. A query run through ##/rest/wikis/query## excludes them for you, which makes the two disagree if only one of them says so. | ||
| 149 | * **Match the extension id on ##property.ExtensionCode.ExtensionClass.id_string##**, not on ##property.ExtensionCode.ExtensionClass.id##. The unsuffixed name is the analyzed text field, so ##*org.xwiki.platform*## also matches an id that merely //contains// it: ##surli:org.xwiki.platform:xwiki-platform-store##, someone's own copy of the Store module, counts as a platform extension for that query and not for ##id_string:org.xwiki.platform*##. The string field is also the only one that can be faceted, which is how the distinct ids are counted. | ||
| 150 | * **Count pages, not documents, with ##doclocale:""##.** The index holds one document per translation, so a wiki whose pages are translated counts high: ##wiki:rendering## answers 119 for 66 pages. The three trees above and the extension pages have no translations, which is why the difference only shows on the rendering wiki. | ||
| 151 | * **##website:*## is not an "is it set" test** - it matches extension pages whose ##website## is an empty string. Match ##ExtensionLD## instead. | ||
| 152 | |||
| 153 | == The metadata behind the numbers == | ||
| 154 | |||
| 155 | * [[MigratedPageClass>>XWikiOrg.documentation-scripts-queries.MigratedPageClass]] - one field, ##newPages##: the references of the pages that replace this one, one per line. Set it on the **legacy** page while converting it. | ||
| 156 | * [[MigratedPageSheet>>XWikiOrg.documentation-scripts-queries.MigratedPageSheet]] - renders those references as a "this content has moved to ..." box on the legacy page, above its remaining content. | ||
| 157 | * The ##unconvertable## tag, added when the page documents something deprecated: there is no place for that content in the new documentation, so the page is neither converted nor mapped, and it leaves the //Left// column without ever becoming //done//. | ||
| 158 | * The ##done## tag, added when the content has been //entirely// moved. The index matches the whole tag value, not the words in it, so a tag such as ##done, reviewed## does not count and a page tagged ##knowledge base## is not found by ##knowledge##. Case is the one thing that does not matter: ##Done## counts. |
