Skip to content

Added pagination for function export - #105

Merged
abnegate merged 1 commit into
mainfrom
dat-609
Aug 7, 2025
Merged

abnegate merged 1 commit into
mainfrom
dat-609

Conversation

@ArnabChatterjee20k

@ArnabChatterjee20k ArnabChatterjee20k commented Aug 7, 2025

Copy link
Copy Markdown
Contributor

Source

image

Destination

image image

Summary by CodeRabbit

  • Refactor
    • Improved the process for exporting functions and their environment variables to handle large numbers of items efficiently through batch processing and pagination.

@coderabbitai

coderabbitai Bot commented Aug 7, 2025

Copy link
Copy Markdown
Contributor

Walkthrough

The exportFunctions method in src/Migration/Sources/Appwrite.php was refactored to implement paginated fetching of functions. Instead of retrieving all functions in a single API call, it now uses a loop to fetch and process functions in batches, handling pagination and incremental accumulation of related resources.

Changes

Cohort / File(s) Change Summary
Appwrite Function Export Pagination
src/Migration/Sources/Appwrite.php
Refactored the exportFunctions method to replace single-fetch logic with a paginated loop, incrementally fetching and processing functions and their environment variables until all are retrieved.

Sequence Diagram(s)

sequenceDiagram
    participant Appwrite as Appwrite::exportFunctions
    participant API as Appwrite API
    participant Callback as Callback

    Appwrite->>API: List functions (with limit, optional filters, cursor)
    loop While more functions
        API-->>Appwrite: Return batch of functions
        Appwrite->>Appwrite: Convert functions and env vars to resources
        Appwrite->>Callback: Invoke callback with resources
        Appwrite->>API: List next batch (with updated cursor)
    end
Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Poem

🐇
In batches now the functions flow,
Through paginated fields they go.
No longer fetched in just one leap,
They hop in batches, not a heap.
With every loop, more data found—
A rabbit’s joy in leaps unbound!

✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch dat-609

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Explain this complex logic.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai generate unit tests to generate unit tests for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
src/Migration/Sources/Appwrite.php (1)

1357-1367: Query building logic is correct.

The pagination query building follows the established pattern. One minor observation: when filtering by root resource ID, the method adds Query::limit(1) after already adding Query::limit($batchSize), which could result in duplicate limit queries. Consider restructuring to avoid this duplication.

-            $queries = [Query::limit($batchSize)];
-
             if ($this->rootResourceId !== '' && $this->rootResourceType === Resource::TYPE_FUNCTION) {
                 $queries[] = Query::equal('$id', $this->rootResourceId);
                 $queries[] = Query::limit(1);
+            } else {
+                $queries = [Query::limit($batchSize)];
             }
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 34bb7c4 and 73cd271.

📒 Files selected for processing (1)
  • src/Migration/Sources/Appwrite.php (1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📓 Common learnings
Learnt from: ItzNotABug
PR: utopia-php/migration#89
File: src/Migration/Sources/Appwrite/Reader/API.php:64-84
Timestamp: 2025-07-19T08:29:22.290Z
Learning: In the Appwrite API, the default page limit for listing collections is 25 records, so when using cursor-based pagination with Query::cursorAfter(), there's no need to explicitly specify Query::limit(25) as the API will default to this limit.
Learnt from: ItzNotABug
PR: utopia-php/migration#80
File: src/Migration/Sources/Appwrite/Reader/API.php:8-8
Timestamp: 2025-06-28T09:47:11.436Z
Learning: In the Appwrite migration codebase, commented-out Tables service references (import statements and constructor parameters) are intentionally kept for future implementation when the Tables service becomes available in the Appwrite SDK, rather than being dead code that should be removed.
🔇 Additional comments (3)
src/Migration/Sources/Appwrite.php (3)

1352-1356: LGTM! Proper pagination initialization.

The initialization follows the established pagination pattern used throughout this codebase, with correct type annotation and infinite loop structure.


1368-1373: LGTM! Proper early termination handling.

The API call and early return logic correctly handles the case when no functions are available, using appropriate strict comparison and void return.


1375-1411: LGTM! Efficient resource conversion and proper loop termination.

The implementation correctly:

  • Converts both functions and environment variables in a single pass
  • Accumulates related resources for efficient callback processing
  • Uses proper loop termination condition based on batch size
  • Updates pagination cursor appropriately

This approach is more efficient than the previous single-fetch implementation and maintains consistency with other paginated export methods in the codebase.

@abnegate
abnegate merged commit 025b995 into main Aug 7, 2025
4 checks passed
@abnegate
abnegate deleted the dat-609 branch August 7, 2025 04:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants